Getting response code 400 - bad request requesting Deals on API v3

[SAMsan]

Hello,

I'm trying to request deals per Pipeline using API v3 Search  method.

I can get entries at some point.

Using "after"=999 is working but "after"=9999 fails returning 400 - bad request.

Here are the logs :

{limit=100.0, filters=[{propertyName=pipeline, value=571502.0, operator=EQ}], after=999, sorts=[{propertyName=createdate, direction=ASCENDING}], properties=[dealname, amount, dealstage, sign_par, hubspot_owner_id, rdv_vecu, produit_vendu, type_de_produit, statut_dossier, origine, motif_de_contact, raison_des_clotures, lead_brut_lead_net_contacts, url_site]}
----- RETURN -----
UrlFetchApp result: 200.0 {total=10875, paging={next={after=1099}}}


{limit=100.0, filters=[{propertyName=pipeline, value=571502.0, operator=EQ}], after=9999, sorts=[{propertyName=createdate, direction=ASCENDING}], properties=[dealname, amount, dealstage, sign_par, hubspot_owner_id, rdv_vecu, produit_vendu, type_de_produit, statut_dossier, origine, motif_de_contact, raison_des_clotures, lead_brut_lead_net_contacts, url_site]}
----- RETURN -----
UrlFetchApp Failed: 400.0 {"status":"error","message":"There was a problem with the request.","correlationId":"27d1f756-d1ee-4943-a4e4-610611b1a0b6","requestId":"412b9bca-7814-41f4-9fb4-72d1ebb76215"}

Obviously, I won't be able to request all my deals...

[adley_amzovski_]

Hi @Willson

Is there a timeline to fix this limitation? We have updates of 800K contacts to be pulled on a daily basis that we're filtering using the lastmodifieddate param. Do you have a better solution to achieve the data pulls?

[Willson]

Hi @SAMsan 

Thanks for providing this! To confirm, the after parameter is used to access the next page of results and the value entered is the paging.next.after property seen in the previous response.

If the paging.next.after property isn’t provided, there are no additional results to display.

As you're searching Deals in your Pipelines, I would not expect you to be fetching this many results that the paging.next.after value returned is 9999. Can you confirm?

[SAMsan]

@Willson 
I do indeed receive the after parameter once the first limit is reached.

My code loops until the after parameter is absent.

As I request all deals from one pipeline for analytics purpose, request's total items is above 10k.

Thank you for your help

[Willson · Accepted solution]

Hey @SAMsan 

Thanks for your patience here! I was able to dig into this internally and find that this is a known limitation currently, as such, we've updated the new documenation to include this now. 

The new limitation reads:

The search endpoints are limited to 10,000 total results in the response. Attempting to page beyond 10,000 will generate a 400 error.

See here: https://developers.hubspot.com/docs-beta/crm/search

I hope this helps!

[StanBartczak]

Hi @Willson is there no way to only check the number of contacts/deals matching some criteria if there are more than 10k results?

[saps]

Hi @Willson , when will this issue be resolved? While it isn't a big problem for Deals, it is a massive problem for Contacts - the Contacts API (without filterable results) provides too much information from past (years/months) whereas the Search API gives too less. Having more than 10k active Contacts is pretty normal and not an edge case.

[Willson]

Hi @SAMsan 

I just wanted to follow-up here! I am looking into this further from my end and will revert with an update once I have more for you.

Feel free to add any questions you may have in the meantime. Thanks

[Willson]

Hi @SAMsan 

In order to assist, would you be able to forward on the exact Request URL you're currently using and the associated v3 documentation you're referencing too for me? Thanks!

[DZaboronsky]

Hello, are there any updates? This issue is 2 years old and still exists on the search enpoint as far as I can see? (experiencing on the contacts search endpoint). Also this limit is not documented here: https://developers.hubspot.com/docs/api/usage-details#rate-limits under `other limits`. Just wondering what is the correct method to handle big resposnes?

I can see it is in the docs at: https://developers.hubspot.com/docs/api/crm/search but there is no suggestion for work around, and as it looks on this thread this is a known limitiation which is now ~4 years old?

[SAMsan]

Thank you @Willson 

The request url I use is :
https://api.hubapi.com/crm/v3/objects/deals/search?hapikey=**myAPIkey**
I am using this documentation page : 
https://developers.hubspot.com/docs-beta/crm/search