Search API - Batch size not always 10,000

BHarrison-Cook — Tue, 04 Jul 2023 11:56:26 GMT

We have download routines built in SQL Server Integration Services, that are designed to download the all records from certain objects. We do this by the using the Search Endpoint and passing in a body like this:

{"filterGroups": [

{

"filters": [

{

"value": {{User::RecordID}},

"propertyName": "hs_object_id",

"operator": "GTE"

}

]

}

"sorts": [

{

"propertyName": "hs_object_id",

"direction": "ASCENDING"

}

"properties": [

"limit": 100,

"after": <%after%>

}

We call this in a loop, as the Search Endpoint will only return 10,000 records. So we loop through 10,000 and if the record count is < 10,000 we stop the process. If the Record Count is = 10,000, we grab the last ID from the last call and then make another call as above which does hs_object_id >= last ID returned. So in effect, we are looping through in batches of 10,000 until we have returned all ID’s in the object.

We have identified however, that on our Line Items package, this routine was failing to download all IDs. After investigating, I have identified that the Search Endpoint does not always return 10,000 records.

In a test I completed yesterday on our portal regarding the Line Item object:

Batch 1 – Returned 10,000.

Batch 2 – Returned 10,000

Batch 3 – Returned 10,000

Batch 4 – Returned 9997

Our routine then stopped, as the last batch was < 10,000.

However, after tweaking our criteria, I amended it to say < 9900 then continue. And this behaviour was then witnessed:

Batch 1 – Returned 10,000

Batch 2 – Returned 10,000

Batch 3 – Returned 10,000

Batch 4 – Returned 9997

Batch 5 – Returned 9177

In this last test, we then returned all ID.s

My question therefore is, why did Batch 4 return < 10,000 records? My expectation, after reading the HubSpot API Documentation, is that the Search Endpoint will return 10,000 records if there are more records remaining from the query.

I need to understand this behaviour so I can best align our download routines for every object to ensure we do not miss any data. We have many internal reports configured from our downloaded data in our data warehouse, so this is of high importance to ensure we have correct reports been shared with our business.

Does anyone have any thoughts on why this endpoint behaved in this way?

topic Search API - Batch size not always 10,000 in APIs & Integrations

Search API - Batch size not always 10,000