APIs & Integrations

PowerMyAnalytic
Top Contributor

Search the CRM API 10,000 objects limit

I use Search V3 to fetch contacts that created or last modified between a certain date range.

 

The limit of 10,000 objects is not enough because there are some days that the number of modified contacts is ~50,000 - and we talk about date range of one day!!

 

I found this answer https://community.hubspot.com/t5/APIs-Integrations/Error-on-contacts-search-over-10-000-when-using-p...

 

But I think that my use case is basic and a wonderful tool like the Search API should support it.

 

Can you relax this limitation?

5 Replies 5
PowerMyAnalytic
Top Contributor

Search the CRM API 10,000 objects limit

WendyGoh
HubSpot Employee
HubSpot Employee

Search the CRM API 10,000 objects limit

Hey @PowerMyAnalytic,

 

For now, we wouldn't be able to remove the limit entirely and it is likely that any reasonable limit that protects our databases will eventually be hit by a developer working through a large data set. 

 

Having said that, we do understand that this is going to be a common problem.

 

Hence, one workaround that you can page through the dataset in an efficient ways is to add more-specific filters and sorts in order to page through the result set. For example, if they filter by LastModifiedDate and sort by vid then they could select the next page of data by updating the filters to filter by the LastModifiedDate + vid of the last record returned by the search. 

 

Hope this helps!

0 Upvotes
AmitMY
Participant

Search the CRM API 10,000 objects limit

Hi Wendy,

To share some more context,

Right now, we are paginating as efficiently as possible.

Some of our clients have around a million contacts, and if we just check which of them were modified in the past week, we get around half of them.

 

In order to get 500,000 contacts, with a page limit of 100, we need to make 5000 requests to your server. (per user!)

 

This means that we can't give this to our customers in real-time, and instead, have to store everything in our databases.


Instead, we just ask that you could dump the entire contacts, maybe even as a gzip-compressed JSON, which we know is less than 10MB in size.

Doing so will save you, and us, so much on bandwidth, both in terms of infrastructure and in terms of speed of data acquisition. It will also make the data for the client more accurate as we could just query a list of contacts as the client needs them, in real-time.

 

I'd be happy to jump on a call with you or one of your devs to explain our exact strategies, and how supporting this will undoubtedly be smart both for you and for us.

 

Amit

PowerMyAnalytic
Top Contributor

Search the CRM API 10,000 objects limit

@WendyGoh 

Amit forgot to tag you

0 Upvotes
DeepakKataria
Member

Search the CRM API 10,000 objects limit

Hi @PowerMyAnalytic 

 

Did this issue resolved?

0 Upvotes