Unable to fetch more than 100 records from Hubspot API

[DKumar]

We are trying to read records of "Contact"/"Engagement" using Hubspot API. While performing this activity, we are only able to fetch 100 records at a time using this URL--

https://api.hubapi.com/crm/v3/objects/contacts?&limit=100&properties=lifecycle_stages,firstname,last...

and upon increasing the limit we are getting error 400 bad request(refer screenshot).
We also tried exceuting the same API URL repetitively but we are only able to read the first 100 reads.

[lynton · Accepted solution]

@DKumar this depends on the programming language, but any software developer should be able to apply the paging method to their code. If you're looking for off the shelf software to make it easier, check out products from Skyvia, CData, Kingswaysoft, or Stitch Data for various options to extract this data with minimal or no code.

[DKumar]

Hi All,

We are actually trying do an API Integration of HubSpot with our system. For this, we are trying to use a single URL for reading all the records in contact but we are not able to fetch all record in single URL for this we have to update the URL for every 100 Records due to hubspot paging. for this issue we have suggested a solution to look at  third-party API wrappers and tools, such as Kingswaysoft or CData which allows for SQL-like queries against HubSpot without fussing with the underlying API mechanics.  But we don't want to use third-party API wrappers and tools so except this is there any other solution through which we can fetch all contact record in singel URL.

[lynton]

@DKumar multiple api requests using a paging token is a requirement of any rest API, as it should not be expected that a single http request can respond with all records in the system. The response payload would be too large. If you aren't looking to use any third-party API wrappers, you will need to support the paging tokens to loop through all records from the HubSpot API.

[DKumar]

@lynton Thanks for your Response.Could you please explore me that how we can support paging tokens to loop through all records from the Hubspott API. 

[lynton]

Hi @DKumar this is a fairly common method of paging through multiple records in a REST API. As you mentioned, your HubSpot database will grow over time. It's not technically feasible to return all records in a single api call, so HubSpot uses this paging method.

If you're looking for easier ways to paginate and query HubSpot API data, you can look at third-party API wrappers and tools, such as Kingswaysoft or CData which allows for SQL-like queries against HubSpot without fussing with the underlying API mechanics. I hope this helps!

[dennisedson]

@DKumar 

For anything more than 100, you will receive a paging object that will allow you to send a new request with the additional query param.  This will load the next 100.

Looks something like this

"paging": {
    "next": {
      "link": "?after=NTI1Cg%3D%3D",
      "after": "NTI1Cg%3D%3D"
    }
  }

[DKumar]

@dennisedson 

Hi Dennisedson, thanks for taking out time to suggest a solution on this.  While making use of this solution, we will be required to create a new URL whenever we need to fetch the next 100 records. Is there way through which we can use a single URL for fetching 100 next everytime we hit.

[dennisedson]

Hi @DKumar 

You will need to update the url with the new parameter.

[DKumar]

Hi @dennisedson ,

Thanks for your response. We are actually trying do an API Integration of HubSpot with our system. For this, we are trying to use a single URL for reading all the records in contact. The number of records of contacts in HubSpot are expected to keep on increasing. In the suggested solution, we will be required to update the URL for every 100 records. Can you please suggest a solution accordingly, wherein, we can make use of a single URL for fetching all contact records.

[dennisedson]

@DKumar 

To the best of my knowledge, you cannot.  There might be a client library that obfuscates it, but that is the limit.

@lynton could you correct me if I am wrong here?