"Batch Update" issue

[airbyte]

Hey, we've seen that recently, we got a "Batch Update" in the source of many records. Trying to get more than 10,000 records is a problem due to CRM search endpoint limitations, so we can not get over those, as all of them have the same updated date.

• The search endpoints are limited to 10,000 total results for any given query. Attempting to page beyond 10,000 will result in a 400 error.

Is there any workaround for this?

[SteveHTM · Accepted solution]

@airbyte - It's uncler to me what you might be trying to achieve here, but see below.

To overcome some of the hard limits of the API world, the only viable approach is to subset your data in various arbitrary ways - I sometimes split Contact segments by first letter of email or by last digit of the record ID to end up with manageable numbers for any given workflow or update operation.

The thing that catches my eye on your screenshot is the source label of 'HubSpot processing'. I presume the property is 'marketing contact status'? If so,  I had this stange value explained to me a few months ago in a support session. HubSpot has apparently started scanning customer instances for cases of marketing contacts without 'legal basis for processing' being set to any value. Its potentially a kind of enforcement of consent management rules to maintain the HubSpot email sending reputation. Perhaps taking a look at this would alter the nature of your challenge?

Best of luck.

Steve

[Kevin-C · Accepted solution]

Thanks @Jaycee_Lewis 

Hey @airbyte 

Depending on the property and value you're searching for you can use the graphql API to page through virtually all of the supported records in HubSpot. I've found this method to be the most effecient.

You could also use the object batch endpoint, though the page size will be substanctially smaller.

I will also second @SteveHTM solution to segment.

To close, it might help us I dentify a solution if you could provided more context as to why you're using the search endpoint and the type of data you're querying.

Best

[SteveHTM · Accepted solution]

@airbyte - It's uncler to me what you might be trying to achieve here, but see below.

To overcome some of the hard limits of the API world, the only viable approach is to subset your data in various arbitrary ways - I sometimes split Contact segments by first letter of email or by last digit of the record ID to end up with manageable numbers for any given workflow or update operation.

The thing that catches my eye on your screenshot is the source label of 'HubSpot processing'. I presume the property is 'marketing contact status'? If so,  I had this stange value explained to me a few months ago in a support session. HubSpot has apparently started scanning customer instances for cases of marketing contacts without 'legal basis for processing' being set to any value. Its potentially a kind of enforcement of consent management rules to maintain the HubSpot email sending reputation. Perhaps taking a look at this would alter the nature of your challenge?

Best of luck.

Steve

[airbyte]

These are the properties.

And yes I think "Batch Update" process at the end of the month to reset marketing status.

[Jaycee_Lewis]

Hi, @airbyte 👋 Thanks for your question. There isn't a way to get around the search API limitation of 10000 total results. That is a fixed limit that cannot be adjusted or increased. Let's invite some of our community champions to the conversation — hey @SteveHTM @Kevin-C @evaldas do you have any suggestions for @airbyte?

Thank you very much for taking a look! — Jaycee