topic Re: How to fetch more than 10,000 records using Hubspot API in APIs & Integrations

How to fetch more than 10,000 records using Hubspot API

EInc — Tue, 08 Feb 2022 13:03:55 GMT

Hi,

I am using Deal search API to fetch records with some conditions on it, but I found from the docs that search API is limited to 10,000 records and we can't paginate more than that, and also there is request limit of 4 request/second. So I suppose that even I can't get all the 10, 000 records since that would contain 100 API calls.

My Question is there way I can fetch all the available records with any other API with some conditions applied on the properties

Re: How to fetch more than 10,000 records using Hubspot API

TitiCuisset — Wed, 09 Feb 2022 11:11:45 GMT

Hi @EInc

Thank you for reaching out.

I want to tag some of our experts on this - @himanshurauthan @JBeatty @Bryantworks do you have any thought for @EInc on this?

Thank you!

Best

Tiphaine

Re: How to fetch more than 10,000 records using Hubspot API

vkumar34 — Tue, 03 May 2022 09:28:50 GMT

I Have same question !!
Eagerly Waiting for reply.

Re: How to fetch more than 10,000 records using Hubspot API

DPassman — Tue, 30 Jan 2024 13:51:53 GMT

The Search API returns records starting with the lowest object ID, by default. Therefore, in order to get around this, you'd need to implement a check in your code that logs 2 things.

1. The "after" value

2. The last ID

Once the after value reaches 10,000, you'll need to jump to a new loop in your code that resets the after value, and does a seach for record IDs greater than the last ID value from above. Every time you hit the 10000 after value, you'll need to restart your loop.

Re: How to fetch more than 10,000 records using Hubspot API

GLanker — Tue, 05 Mar 2024 20:04:04 GMT

Going off of the previous posts, the way I have solved this is to re-index instead of paging thorugh. To do this, on the "get" I am sorting by the ID ascending, and storing the last id (array[99]) since I am grabbing 100 at a time. On the next time through I am then checking for hs_object_id to be greater than that stored last id. This is my pseudo code to make this possible and it is pretty seamless

"limit": 100,

"after": "0",

"properties": [ "firstname", "email", "lastname"],

"filterGroups":[

{

"filters":[

{

"propertyName": "lastmodifieddate",

"operator": "GTE",

"value": "{last date to get from}"

{

"propertyName": "hs_object_id",

"operator": "GT",

"value": "{last id stored value (default to 0)}"

}

]

}

"sorts": [

{

"propertyName": "id",

"direction": "ASCENDING"

}

]

}