Offer API calls in small batches with delay

[EvaUR]

Hi there!

I'm trying to find a solution for the TEN_SECONDLY_ROLLING error with one of our workflows. It was suggested to me to create a function that processes our calls in smaller batches with a delay of 10 or more seconds. I've written a code that doesn't seem to run into any errors, but also doesn't give the delay I'd expect. I'm still seeing 200+ calls in a 10 second window.

What would be a better solution for this issue than the code below?

async function processInBatches(assResults, hubspotClient) {

    const batchSize = 150;

    const totalItems = assResults.results[0].to.length;

    const numBatches = Math.ceil(totalItems / batchSize);

    let startIndex = 0;

    let endIndex = 0;

    for (let i = 0; i < numBatches; i++) {

        endIndex = Math.min(startIndex + batchSize, totalItems);

        const batchItems = assResults.results[0].to.slice(startIndex, endIndex);

        await processBatch(batchItems, hubspotClient);

        startIndex = endIndex;

        if (i < numBatches - 1) {

            await delay(10000); // Delay for 10 seconds between batches

        }

    }

}

function delay(ms) {

    return new Promise(resolve => setTimeout(resolve, ms));

}

[dsmarion · Accepted solution]

I use a sleep function that works for me:

    static async sleep(milliseconds) {
        const date = Date.now();
        let currentDate = null;
        do {
            currentDate = Date.now();
        } while (currentDate - date < milliseconds);
    }

[SteveHTM]

Just bear in mind that custom code workflow steps can only last a max of 20 wallclock seconds - including sleep time. So I have found that a delay function has to be teased apart from the actual API call functions to portion out your API call resources into usuable chuncks.

Steve

[EvaUR]

Hi SteveHTM, thank you for your reply. I'm a little new to working with API, could you explain what the difference is in seperating the delay from the other call functions?

[SteveHTM · Accepted solution]

A classic case for me is a workflow (that includes API calls in custom code workflow steps) triggered by an import lets say. If the import contains many records, I try and spread the delay of the workflow trigger for groups of contacts based on a "random" number for each contact.

In fact I often use the HubSpot record ID as a randomizer - the last digit for example can help set up 10 options of programmable delay, which is often enough to avoid the API rate crunch.

Hope this is helpful.

[EvaUR]

This helped me find the right solution for us, thank you!

[dsmarion · Accepted solution]

I use a sleep function that works for me:

    static async sleep(milliseconds) {
        const date = Date.now();
        let currentDate = null;
        do {
            currentDate = Date.now();
        } while (currentDate - date < milliseconds);
    }

[EvaUR]

Thank you for sharing. I'm going to try this out!