Hubspot Rate Limit issue with Company to Contact Association

[JakeAGill]

Just to preface no identifying data is held in our DB. We like to keep a copy of our Hubspot data set in a database, the process is as follows:

Get Companies at: 

api.hubapi.com/companies/v2/companies/paged

Get Contacts at:

api.hubapi.com/contacts/v1/lists/all/contacts/all

 Now for maxiumum accuraccy I request each companies list of Contacts IDs using:

api.hubapi.com/crm-associations/v1/associations/Company ID/HUBSPOT_DEFINED/2

For a good 6+ momths I have been requesting each Companies Contact Associations 

I have been requesting these using threading so increase the speed of requests since we have over 10,000 and sequential requesting takes upwards of an hour to complete.

Since friday I am now receiving 429 error codes, so I assumed I need to reduce the # of threads so I request less frequently to fit into the 100/10s burst rate limit. However now it is taking 30+ minutes to complete these requests and i still receive a significant number of 429 error codes.

Is there a better way to be getting these associations out of Hubspot because the current method is cumbersome.

Is there a reason why a Company cannot come with their associated Contact IDs in the companies endpoint I pasted above?

Why am I now suddenly exceeding the rate limit when for a long period of time I wasn't.

If this is the only way to get the data then I am concerned that this is not scalable as we continue to expand our dataset these functions will only become slower as we need to request more associations for more companies.

This database is a critical part of our reporting functionality and so cannot be replaced for Hubspots internal systems.

[Derek_Gervais · Accepted solution]

Hey @JakeAGill ,

In order to really understand the rate limit errors you're running into, it'd be useful to know more about the errors. Are you logging the error response bodies? That should include which limit you're hitting (burst or per-day).

Additionally, you might consider using the rate limit headers that HubSpot returns with each request (see below). The X-HubSpot-RateLimit headers include how much of your daily/burst allowance you have left, and when the brust allowance will reset. Adjusting the requests you're making based on the information in these headers can help optimize your API consumption and avoid 429 errors. Here's the developer doc on these headers: https://developers.hubspot.com/docs/faq/working-within-the-hubspot-api-rate-limits

All that said, you can significantly reduce the number of API calls you're making by requesting the `associatedCompanyId` property when pulling all of your contacts, like this: ?property=associatedCompanyId Keep in mind, this property is read-only; the CRM Associations API must still be used to update the contact's associations. 

[JakeAGill]

Hey I've been exceeding the 10s rate limit. I did not know that contacts had an avaliable property called associatedCompanyID, that solves all of my problems thank you so much 😄