topic Re: Errors when enrolling many objects into a custom code workflow step in APIs & Integrations

Errors when enrolling many objects into a custom code workflow step

WaydeChristie1 — Tue, 01 Feb 2022 03:08:10 GMT

Hi folks,

I have a deal-based workflow which includes a custom code step.

The custom code searches for a custom object id, then copies a property from the custom object to the deal.

We get hundreds of new deals which are synced twice per day.

When many deals are enrolled in the workflow at once, I'm seeing the custom code step return HTTP 400 errors, and only a small percentage of the enrolled deals actually complete the code step successfully.

If I manually enrol any of the deals which failed individually, the code step completes successfully.

I'm thinking this is likely due to Search API limits, but the error reporting isn't helping me.

I've tried adding a delay step before and after the custom code, but no cigar.

I'm thinking I may need to add a delay in the actual code step itself.

Any suggestions here?

Thanks in advance.

Re: Errors when enrolling many objects into a custom code workflow step

WaydeChristie1 — Tue, 01 Feb 2022 08:20:12 GMT

I think my problem here is that many deals are enrolling at the same time, and there's no way for me to sequentially process them several seconds apart to avoid hitting the 4 per second search API limit.

If I add a delay step, or add a pause in my code step, the workflow still tries to process many search requests at the same time.

What I need is a way to process each deal in the workflow several seconds apart.

How would I go about that?

Re: Errors when enrolling many objects into a custom code workflow step

webdew — Fri, 04 Feb 2022 08:37:36 GMT

Hi @WaydeChristie1 ,

Why don't you trigger your workflow directly when a new deal is created so that at one instance it will run only for that one particular deal.

Hope this helps!

If we were able to answer your query, kindly help the community by marking it as a solution.
Thanks and Regards.

Re: Errors when enrolling many objects into a custom code workflow step

WaydeChristie1 — Fri, 04 Feb 2022 23:19:07 GMT

Thanks.

Hundreds of deals are enrolling at the same time which is making the search API hit its limit of 4 requests per second.

I need a way for the workflow to *sequentially* processed the deals with a delay between each run of the workflow.

Re: Errors when enrolling many objects into a custom code workflow step

JacquelineOng — Mon, 23 May 2022 08:40:41 GMT

Commenting here as I had a similar case with a customer:

From checking in with our product team, we confirmed that there isn't a way of staggering workflow enrollments at the moment (i.e. if many objects are being enrolled into the same time, they can't be processed or staggered with a few seconds or minutes apart).
I'd suggest posting a new idea for this and sharing your use case with us over in the Ideas forum here which will hopefully let our product team know.

In the meantime, the only way around this is to adjust the process you're using to enroll objects into the workflow. Some possible options:

It sounds like you might be enrolling deals based on an automatic enrollment trigger set up in the workflow, so you could instead remove the trigger and manually enroll contacts. This way, you can choose how many objects you want to enroll at once. If you want to continue using an enrollment trigger, being more specific with the trigger and basing it off different property values might help.
You can also try to experiment with your action execution settings - see documentation here. If the workflow only executes actions during specific time frames throughout each day, you may be able to reduce the amount of objects attempting to go through the workflow actions simultaneously.

That said, while the above are some suggestions, there isn't any single "best" option here as this will depend on your specific needs. Hope this helps to point you in the right direction!

If this post helped, please mark it as a solution to help more users.

Re: Errors when enrolling many objects into a custom code workflow step

PBaxter — Fri, 24 Jun 2022 14:21:53 GMT

Hey Wayde - did you ever find a solution for this? I have a similar challenge right now. Thanks, Phil

Re: Errors when enrolling many objects into a custom code workflow step

WaydeChristie1 — Sun, 26 Jun 2022 23:05:03 GMT

Hi Phil - unfortunately not. Until Hubspot builds a way to sequentially step objects through a workflow, I don't think it will be possible.

Re: Errors when enrolling many objects into a custom code workflow step

PBaxter — Wed, 10 Jan 2024 10:57:32 GMT

Zapier now has a nicely implemented delay method based on a very simple queue - essentially all enrolled objects get pushed onto a queue and then released every 'x' seconds/minutes. It'd be good if HubSpot could pick up something similar. It's getting more acute as an issue as leveraging the OpenAI API for example is super difficult without better control of rates

In the meantime, I'm now using that zapier delay/queue step as a work-round in a number of workflows.

Re: Errors when enrolling many objects into a custom code workflow step

HubSpot_Corey — Tue, 16 Jul 2024 03:20:48 GMT

Hi Everyone,

I know this post is a bit older now, but I thought I'd add a solution for anyone that's having similar issues.

The issue with using any HubSpot delays is that if all the objects are enrolled at the same time, for example when an import is done they will hit the delay stage at the same time, therefore they will all still hit your custom coded step at the same time, not resolving the limit issue.

What I suggest doing is an exponentially increasing randomised delay using setTimeout(). I've put an example of how this can be done below:

{ let timeout = 5000; const maxTries = 5; let outcome = 0; let data; for (let i = 1; i <= maxTries; i++) { const response = await fetch(url, options); if (response.status === 200) { data = await response.json(); outcome = 200; break; } else { console.log(`Attempt ${i} failed. Retrying in ${timeout}ms...`); await new Promise(resolve => setTimeout(resolve, timeout)); timeout *= 3*Math.random(); } } if (outcome !== 200) { throw new Error(`Max retries reached. Unable to fetch data.`); } }

As you can see, everytime the response fails, it multiples the delay anywhere from 0-3 times and retries up to 5 times before erroring.

For the timing I would suggest finding out what you're largest import/creation event would be and see what timeframe this would need to be split over to be within the rate limit.

Re: Errors when enrolling many objects into a custom code workflow step

Clare_HS — Wed, 25 Sep 2024 13:22:48 GMT

Echoing that this is an older post, btu the problem seems to persist. I finally decided to just add a multiple custom code steps with ONLY randomly-generated wait times. So the whole step just looks like this:

exports.main = async (event) => { // The number io line 6 represents time in milliseconds. // So 15000 = 15 seconds. You can go up or down, // but it will time out automatically if you go over 20000. let timeInMs = Math.random() * (15000); let test = function (){ console.log('Waited ' + timeInMs) }; setTimeout(test,timeInMs); }

And I added in three of these at the beginning of my workflow. Nothing fancy, at all, but it seems to smooth the rough edges enough.

Note: this is technically commented above, but I'd stay under 15000 or so in real-world applicaitons. 15000 is probably about as high as I'd go since custom code has to execute in 20 seconds or less.