topic Re: API Speed while pulling data in APIs & Integrations

API Speed while pulling data

SKumar00 — Thu, 04 Sep 2025 15:05:42 GMT

If I'm hitting the entire Deals, Company, Contacts, and Association API to get daily refreshed data, why does it take more than 2 or 3 hours? We don't have millions of records. Is there a way to speed up the process?

Re: API Speed while pulling data

Victor_Becerra — Thu, 04 Sep 2025 19:41:23 GMT

Hi @SKumar00
Thank you for reaching out to the Community!
I'd like to invite some community members who are subject matter experts to join this conversation.
@SteveHTM @h-recker @evaldas - Would you be able to share any insights on this? Your expertise would be greatly appreciated.
Best,
Victor

Re: API Speed while pulling data

SteveHTM — Fri, 05 Sep 2025 18:30:52 GMT

@SKumar00 - can I ask if you are considerinbg the API rate limits for your functionality? Cases where rate limits and back-off functions are invoked can make a big difference to workflow performance.

Hope this is helpful.

Steve

Re: API Speed while pulling data

SKumar00 — Mon, 08 Sep 2025 17:05:50 GMT

@SteveHTM, Currently, I am not considering rate limits while fetching this data. Could you suggest a better workflow to improve the data fetching?

Re: API Speed while pulling data

SteveHTM — Mon, 08 Sep 2025 17:37:35 GMT

The first thing to understand is whether rate limits are being hit as I suspect - I could be way wrong. You will need to look at the logs of your private app to see if 429 codes get retured with any frequency.

If this is the case, then the typical response is to try and use batch API calls and/or throttle the app process by dividing your data into logical sub sets of some type.

It's hard to say more with the data you have shared.

Steve

Re: API Speed while pulling data

SKumar00 — Tue, 09 Sep 2025 09:29:20 GMT

In our private app logs, there are no 429 error codes appearing, which suggests we are not hitting any rate limits; these would be displayed in the private apps. Let me explain the issue we are facing: We have over 50,000 deals, each associated with companies, contacts, and line items. However, retrieving the associated data through the association API is taking around three hours, which is significantly longer than it should.

Re: API Speed while pulling data

SteveHTM — Tue, 09 Sep 2025 09:57:35 GMT

@SKumar00 - OK, my theory about rate limits was incorrect. But 50k records seems like it might involve a lot of API calls - I'd probably be trying to benchmark smaller segments of data to understand if your code scales properly to these large numbers.

Good luck!

Steve

Re: API Speed while pulling data

SKumar00 — Wed, 10 Sep 2025 12:37:32 GMT

@SteveHTMCan you share some best practices for fetching these records? My ultimate goal is to refresh all my HubSpot data every data.

Re: API Speed while pulling data

Victor_Becerra — Thu, 18 Sep 2025 18:41:20 GMT

Thanks for keeping the conversation going here, everyone!

@SteveHTM really appreciate the detailed troubleshooting steps you’ve shared around rate limits, batching, and benchmarking smaller subsets — those are solid practices to test.

@SKumar00 thanks for clarifying the scale and sharing that you’re not seeing any 429 errors — that context is super helpful for narrowing things down.

To get a few more eyes on this, I’d also like to tag in @evaldas @h-recker and @alyssamwilie — would love to hear if you’ve come across something similar in your work and how you approached it. If you have seen similar behavior or found additional workarounds, feel free to jump in and share!

Thanks, Victor

Re: API Speed while pulling data

h-recker — Mon, 22 Sep 2025 19:18:12 GMT

Some thoughts. From my experience, it's almost always a mix of rate limits on a public API + pulling too much data the slow way.

What could be slowing you down:

Per-endpoint rate limits. The CRM Search endpoints are capped at ~5 requests/sec per account, and 200 records/page. If you’re paging through full tables, that alone can stretch into hours.
General request limits. Most public API traffic is also constrained to ~100 requests per 10s per account (OAuth/private app), so lots of small calls get throttled.
Associations are heavy. Grabbing associations one record at a time is a time sink; make sure you're batching reads.

Some potential solutions:

Switch from “full pulls” to true syncs.
Use the CRM Search API with a filter on hs_lastmodifieddate (a.k.a. updatedAt) ≥ last run. Page at 200 rows/page (the max) and only request the properties you need. This should cut payload and page count.
Consider batch reads if you're not already using them.
Run objects (Deals, Contacts, Companies) in parallel workers.

I know 50,000 deals doesn't seem like a million, but unless you're on a dedicated API there are things happening that are completely out of your control. Seems like you've put a lot of time and effort into this, but I'd definitely recommend a 3rd party solution to make the pulls for you. Maintenance plus adding new field / object requirements in the future could really be a huge time sync. I do work at Coefficient - which offers a 2-way sync between HubSpot and Sheets/Excel on HubSpot's marketplace. And we're a relatively inexpensive solution that could solve this problem for you and future problems overnight.

Re: API Speed while pulling data

RubenBurdin — Wed, 22 Oct 2025 21:19:32 GMT

Hi @SKumar00, this usually comes down to doing full-table pulls and N+1 association calls.

If you can switch to deltas, use the CRM Search API with a last-modified filter and request only the properties you truly need; that cuts pages and payload drastically (https://developers.hubspot.com/docs/api-reference/search/guide )

For associations, avoid per-record reads and hit the v4 Batch Read endpoint to fetch links for up to 1,000 IDs per request, then join on your side; that’s the biggest win on 50k deals with companies/contacts/line items
(https://developers.hubspot.com/docs/api-reference/crm-associations-v4/guide )

Run each object family in parallel workers, but serialize association batches to stay predictable. If you must do a daily “full refresh,” shard by updated window (e.g., hourly buckets) to keep batches warm and resumable.

One clarifying point: do you actually need a complete daily rebuild, or would near-real-time deltas plus a weekly backfill meet the requirement? Where native connectors stop short, Stacksync fills the gap with real-time, bidirectional sync so your workflows and reports stay trustworthy.