topic Re: Headers: Secondly Limit in APIs & Integrations

Headers: Secondly Limit

EIbbetson — Sun, 04 Jun 2023 14:36:58 GMT

X-HubSpot-RateLimit-Secondly-Remaining

How do I get this value? on

/crm/v3/objects/contacts/search

The Create/Update contact batches have it, but my issue is on the search as this is individual per every record.

My headers on Axios as recommended to use here:

https://community.hubspot.com/t5/APIs-Integrations/Wrapped-API-NodeJS-Headers/td-p/707764

This endpoint does not show Secondly Limit Remaining.

I am having to wait to hit it, then increase the timer or reduce records before it hits the timer.

Below is my headers from my call prior to hitting the limit error.

AxiosHeaders {
date: 'REDACTED',
'content-type': 'application/json;charset=utf-8',
'content-length': '24',
connection: 'close',
'cf-ray': '7d1ff7430d474139-LHR',
'cf-cache-status': 'DYNAMIC',
'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
vary: 'origin, Accept-Encoding',
'access-control-allow-credentials': 'false',
'x-envoy-upstream-service-time': '50',
'x-evy-trace-listener': 'listener_https',
'x-evy-trace-route-configuration': 'listener_https/all',
'x-evy-trace-route-service-name': 'envoyset-translator',
'x-evy-trace-served-by-pod': 'iad02/hubapi-td/envoy-proxy-598c95b5b7-ltr78',
'x-evy-trace-virtual-host': 'all',
'x-hubspot-correlation-id': '46653b25-1ec7-4b68-a63e-2ed45bc5bfd8',
'x-hubspot-ratelimit-daily': '500000',
'x-hubspot-ratelimit-daily-remaining': '499851',
'x-request-id': 'ee0dc67a-d6bd-443a-96cd-80c8f9f5ea41',
'x-trace': '2BBEDAE212973A0E093B676CA8A25C66FB635D802F000000000000000000',
'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=Z22pQqwg7p1CWLI6hKBrcSpzAA14OTbxDMgYpQXO9ctyPQCdqEBdOK8Mkd5%2FtHF1NDewyWOWMnsTMjbP%2Fpv7RRPsdIhXS9EAeqdVNY6Nj%2B674Ezq1Z%2FWBesHcgHUy9Xe"}],"group":"cf-nel","max_age":604800}',
nel: '{"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}',
server: 'cloudflare',
'alt-svc': 'h3=":443"; ma=86400'
}

Re: Headers: Secondly Limit

Teun — Mon, 05 Jun 2023 06:53:55 GMT

Hi @EIbbetson ,

I had the exact same issue, there is a reason tho. The 'CRM Search' endpoints have a different limit than the other API endpoints.
They have a burst limit of 4 requests per 1 second and do not return the secondly limit that is used for the other endpoints. I am not sure if the search endpoints also count towards the general burst limit, but you need to configure a different rate limiter for the search endpoints.

https://developers.hubspot.com/docs/api/crm/search#limitations

Re: Headers: Secondly Limit

EIbbetson — Mon, 05 Jun 2023 07:13:12 GMT

Thanks Teun. I am using the Secondly Limit on the Batch endpoints so it is only this one. I think the error it sends "You have reached your secondly limit" confused me. I have now limited it to wait 1 second every 3 records.

Re: Headers: Secondly Limit

Teun — Mon, 05 Jun 2023 07:14:04 GMT

I was so confused as well! Glad you could solve it.

Re: Headers: Secondly Limit

dthreatt — Fri, 30 Aug 2024 07:00:55 GMT

No matter what I do I get SECONDLY rate limits. I have been trying to fix this for about 18 months and so frustrated. HubSpot wont even reply to my tickets anymore about it. I even went to inbound to get an answer and zero zilch!!! I even tried random delays from 3-5 seconds between batches of 1. I even tried the bottleneck limiter. Seems like the workflow runs multiple parallel processes which we have no control over. I refuse to pay for a tool that does this as in integration when it works as a workflow but i keep getting rate limited.

```

const hubspot = require('@hubspot/api-client');

const Bottleneck = require("bottleneck");

// Rate limiting configuration

const DEFAULT_LIMITER_OPTIONS = {

minTime: 2000,

maxConcurrent: 3,

id: "hubspot-client-limiter",

};

// Create a Bottleneck limiter

const limiter = new Bottleneck({

minTime: DEFAULT_LIMITER_OPTIONS.minTime,

maxConcurrent: DEFAULT_LIMITER_OPTIONS.maxConcurrent,

});

// Creating the HubSpot client

const hubspotClient = new hubspot.Client({

accessToken: process.env.SCTMAT, // Assuming the accessToken is stored in an environment variable

});

// Wrap HubSpot client's method to respect rate limits

const wrappedSearchApi = limiter.wrap(hubspotClient.crm.objects.searchApi.doSearch.bind(hubspotClient.crm.objects.searchApi));

const wrappedBatchApi = limiter.wrap(hubspotClient.crm.associations.batchApi.create.bind(hubspotClient.crm.associations.batchApi));

exports.main = async (event, callback) => {

console.log('hs_object_id ' + event.inputFields['hs_object_id']);

console.log('dnb_naics_industry_code ' + event.inputFields['dnb_naics_industry_code']);

const naicsCode = event.inputFields['dnb_naics_industry_code'];

const objectType = "2-23011085"; // Custom object type ID

const searchRequest = {

query: naicsCode,

limit: 1,

properties: ["naics_code"],

filterGroups: [{

filters: [{

propertyName: "naics_code",

operator: "EQ",

value: naicsCode

}]

};

try {

const searchResponse = await wrappedSearchApi(objectType, searchRequest);

if (searchResponse.results.length === 0) {

console.log("No custom object found with the specified NAICS code.");

return;

}

// Assuming each result needs to be associated with the same company ID

const companyId = event.inputFields['hs_object_id'];

await processInBatches(searchResponse.results, companyId);

} catch (error) {

console.error("Error:", error);

}

};

// Function to create a random delay between 3 and 10 seconds

function randomDelay() {

// Calculate a random number of milliseconds between 3000 and 10000

const delay = Math.random() * (6000 - 3000) + 3000;

return new Promise(resolve => setTimeout(resolve, delay));

}

async function processInBatches(results, companyId) {

for (let i = 0; i < results.length; i++) {

// Wait for a random delay between 3 and 10 seconds before processing each batch

await randomDelay();

const batch = results.slice(i, i + 1); // Create a batch of 1

await createAssociations(batch, companyId);

// The existing call to limiter.schedule ensures rate limiting

await limiter.schedule(() => Promise.resolve());

}

async function createAssociations(batch, companyId) {

const inputs = batch.map(customObject => ({

_from: { id: customObject.id },

to: { id: companyId },

type: "naics_codes_to_company" // Adjust the association type as needed

}));

try {

const associationResponse = await wrappedBatchApi("2-23011085", "Companies", { inputs });

if (associationResponse.status === "COMPLETE" && associationResponse.results.length > 0) {

console.log(`Batch association complete for company ID ${companyId}`);

} else {

console.log("Association failed", JSON.stringify(associationResponse.errors));

}

} catch (error) {

console.error("Error creating associations:", error);

}

} ```

Re: Headers: Secondly Limit

kennedyp — Tue, 03 Sep 2024 18:28:05 GMT

Hey @dthreatt -- thanks for your contribution here. I understand you have been experiencing this issue for quite some time, so I definitely can empathize with your frustration. I appreciate you bringing it to our attention. You noted that Support is not replying to your tickets about it, though I do see that you currently have an open ticket with our Support team, so hopefully they can provide some helpful context despite this being custom code!

I know you are a developer as well, though I thought I'd provide some additional resources as well for context. If you're running into the TEN_SECONDLY_ROLLING limit, you should throttle the requests that your app is making to stay under that limit.

In addition to throttling the requests, or if you're running into the daily limit, check out the suggestions outlined in our documentation here: https://developers.hubspot.com/docs/faq/working-within-the-hubspot-api-rate-limits

That said, I am happy to tag in some of our experts to see if they can provide some additional context. @Teun, @SteveHTM, @GRajput can you provide some insight or recommendations for @dthreatt to avoid hitting this limit with their custom-coded workflow action? Thank you!

Best,
Kennedy

Re: Headers: Secondly Limit

SteveHTM — Tue, 03 Sep 2024 18:49:04 GMT

@dthreatt - I encourage you to think about smearing API calls across object segments as well as the time delay tactics. Each object instance of a workflow creates a parallel process, but API calls across those workflow threads are summed up for the purpose of the limit count.

So when I set up a potentially large set of workflows that use API calls somewhere within the logic, I consider how I would segment the onject list into batches that limit concurrent calls. A trivial eample - all contacts with an email address strarting with 'b' get their workfow instances started 10s after those starting with 'a' - and so on.

Hitting these API calling limits can be frustrating, but they are such fantastic tools that figuring out how to work within the system limits is always worthwhile.

Good luck,

Steve

Re: Headers: Secondly Limit

dthreatt — Fri, 06 Sep 2024 16:40:56 GMT

Its not a 10 second limit. The workflow is running multiple concurrent threads not a single one. We as devs do not have control over you guys executing mulptiple threads of the same actions. So its not executing them in a series...their are parrallel processes going. I am looking at the time stamps and can tell. There is no way to limit that from what I can find in your documentation.

Re: Headers: Secondly Limit

SteveHTM — Fri, 06 Sep 2024 17:15:47 GMT

@dthreatt - please think of myself and the other contributors to this forum as fellow developers wrestling with pactical HubSpot deployment issues - just like yourself.

Steve