Workflow Custom Code Action Large Load Retries

[ah8]

Hi,
I have about 175,000 contacts about to go through a workflow and have a custom code action run on them. I've been testing it with 10,000, and I find that I keep seeing, "Custom code wasn't able to execute, but will retry soon" in the Information Event field. That should be fine, but after waiting quite a while, I notice that a lot of the fields that should have been filled in haven't. That is, some didn't make it through the workflow.

Any suggestions on how to do it?

Thank you in advance.

[SteveHTM · Accepted solution]

@ah8 - my sympathies on this. I have had several mysterious occurences of similar hang-ups when dealing with large volumes of custom code workflow invocations. My suspicion is that the interface to AWS Lambda behind the scenes can get bogged down. That's not much help in itself, but running through all your records is likely to require a lot of batch subdivision and speading over longer timescales. At least that's the only thing that's worked for me.

Steve

[GrowthArchitect]

Its because your API hits its secondary limit, generally this is when you first turn it on and it has to bulk update all records, a way round this is to have the code update a value once the code has ran succesfully, you then filter the record using a branch to determine whether this field has been updated, if it has you take no action, if it hasn't set a 1 minute timer and rerun the custom code again, I often have to do this accross multiple branches and delays, eventually the code will work its way through, its certainly not an ideal approach, and you need to be miondful that other custom code or API applications that are using your API may not work correctly whilst you are running this type of script, so I would reccomend applying a "downtime" session during this period if possible. how many branches you need depends on how much the API is overwhelmed, in my experience, its not difficult to hit the API limit, especially with functions that require multiple objects. A better approach is to partition the data and run them through so that the API can handle the requests. Its challenging to do this in custom code as HubSpot workflows with Custom Code Actions do not natively support such pausing and batching so you wuld have to run the workflow continuously and write records into an external queue (e.g., AWS Lambda, Google Cloud Functions, or a local server) but if you can afford the downtime, you can avoid this, just ensure that you are logging when the code has been ran succesfully to quality check your output. 

[SteveHTM · Accepted solution]

@ah8 - my sympathies on this. I have had several mysterious occurences of similar hang-ups when dealing with large volumes of custom code workflow invocations. My suspicion is that the interface to AWS Lambda behind the scenes can get bogged down. That's not much help in itself, but running through all your records is likely to require a lot of batch subdivision and speading over longer timescales. At least that's the only thing that's worked for me.

Steve

[dsmarion · Accepted solution]

If your custom code does not need to return anything to the workflow you could use a webhook action instead where you simply log the webhook in a database or queue system and then use scheduled tasks to process them a batch at a time.  If your custom code must execute and then return something to the workflow for further processing this becomes a bit of a nightmare imo.