topic Re: offset / search api / more than 10000 results in APIs & Integrations

offset / search api / more than 10000 results

stheo — Tue, 20 Feb 2024 08:21:40 GMT

Hello HubSpot Community,

In our business case, we encounter a need to access more than 10,000 results in our search API calls. Presently, we generate reports based on the last modified date of deals, updating our database daily with changed deals.

The challenge arises when the number of updated deals exceeds 10,000 on certain days, causing us to miss updates for more than 10,000 deals.

We propose introducing an offset parameter in the API call, where the limit and after parameters currently work only up to 10,000 results. Below is an example of the API call we envision:

URL: https://api.hubapi.com/crm/v3/objects/deals/search

{ "limit": 100, "after": 10000, "sorts": [""], "properties": [ "manual_delivery_ticket", "hs_pipeline", "erp_contract_type", "actual_delivery_date" ], "filterGroups": [ { "filters": [] } ] }

However, this approach doesn't resolve the issue as it exceeds the 10,000 limitation.

Are we overlooking any existing solutions to address this issue?
If not, is there a possibility of adding an offset feature to the API? We understand the need for limitations but seek the ability to iterate through multiple calls to access all updated results effectively.

Re: offset / search api / more than 10000 results

Jaycee_Lewis — Tue, 20 Feb 2024 20:51:28 GMT

Hi, @stheo 👋 You raise a valid point.

1. You are not missing an option. The documentation states the limit (as you noted) but there is not a workaround or current way to extend this limit

2. Adding your suggestion to the Ideas Community and getting some upvotes is our best move for getting your suggestion in front of the Product team. If you add a link back to your Idea post, I'll give it an upvote to help get the ball rolling

Best,

Jaycee

Re: offset / search api / more than 10000 results

stheo — Wed, 21 Feb 2024 09:39:19 GMT

Hi @Jaycee_Lewis ,

Thanks so much for the reply and your suggestion.

I just posted in ideas community here you will find the link.

BR,

Spyros

Re: offset / search api / more than 10000 results

dannio — Thu, 22 Aug 2024 06:56:47 GMT

@stheo bit late to the party, but my idea to get around this is to do an initial search, then every time the intial 100 has been processed, take the IDs of those 100 and make a filter to not include them in the next search.

i.e after the first 100, add in:

"filterGroups": [ { "filters": [ { "propertyName": "hs_id", "operator": "NEQ", "values": [1, 2, 3, 4, 5] } ] } ]

Then keep adding to the values array every loop. Would that work?

Re: offset / search api / more than 10000 results

dannio — Thu, 22 Aug 2024 10:29:56 GMT

Actually, sorry nevermind. There's a limit to how many you can have in a NOT_IN filter and also the initial request would be too long to process anyway.

I did however run into this:

https://community.hubspot.com/t5/APIs-Integrations/How-to-fetch-more-than-10-000-records-using-Hubspot-API/m-p/571539#:~:text=Once%20the%20after%20value%20reaches,need%20to%20restart%20your%20loop.

So the idea is to use the object ID as a marker

{

"propertyName": "hs_object_id",

"operator": "GT",

"value": "{last id stored value (default to 0)}"

}

and grab the next set of 10000 from that (assuming you're grabbing in ascending order of ID rather than any other search)