topic Re: Unable to fetch more than 100 records from Hubspot API in APIs & Integrations

Unable to fetch more than 100 records from Hubspot API

DKumar — Fri, 26 Mar 2021 05:30:19 GMT

We are trying to read records of "Contact"/"Engagement" using Hubspot API. While performing this activity, we are only able to fetch 100 records at a time using this URL--

https://api.hubapi.com/crm/v3/objects/contacts?&limit=100&properties=lifecycle_stages,firstname,lastname,email,mobilephone,hubspot_owner_id,company,mql_time_stamp,notes_last_updated,fedid,associatedcompanyid,hubspotscore,jobtitle,address,city,state,&hapikey=apikey

and upon increasing the limit we are getting error 400 bad request(refer screenshot).
We also tried exceuting the same API URL repetitively but we are only able to read the first 100 reads.

Re: Unable to fetch more than 100 records from Hubspot API

dennisedson — Fri, 26 Mar 2021 18:31:20 GMT

@DKumar

For anything more than 100, you will receive a paging object that will allow you to send a new request with the additional query param. This will load the next 100.

Looks something like this

"paging": { "next": { "link": "?after=NTI1Cg%3D%3D", "after": "NTI1Cg%3D%3D" } }

Re: Unable to fetch more than 100 records from Hubspot API

DKumar — Thu, 01 Apr 2021 06:52:20 GMT

@dennisedson

Hi Dennisedson, thanks for taking out time to suggest a solution on this. While making use of this solution, we will be required to create a new URL whenever we need to fetch the next 100 records. Is there way through which we can use a single URL for fetching 100 next everytime we hit.

Re: Unable to fetch more than 100 records from Hubspot API

dennisedson — Mon, 05 Apr 2021 20:54:32 GMT

Hi @DKumar

You will need to update the url with the new parameter.

Re: Unable to fetch more than 100 records from Hubspot API

DKumar — Wed, 07 Apr 2021 06:52:58 GMT

Hi @dennisedson ,

Thanks for your response. We are actually trying do an API Integration of HubSpot with our system. For this, we are trying to use a single URL for reading all the records in contact. The number of records of contacts in HubSpot are expected to keep on increasing. In the suggested solution, we will be required to update the URL for every 100 records. Can you please suggest a solution accordingly, wherein, we can make use of a single URL for fetching all contact records.

Re: Unable to fetch more than 100 records from Hubspot API

dennisedson — Wed, 07 Apr 2021 18:28:38 GMT

@DKumar

To the best of my knowledge, you cannot. There might be a client library that obfuscates it, but that is the limit.

@lynton could you correct me if I am wrong here?

Re: Unable to fetch more than 100 records from Hubspot API

lynton — Wed, 07 Apr 2021 19:06:44 GMT

Hi @DKumar this is a fairly common method of paging through multiple records in a REST API. As you mentioned, your HubSpot database will grow over time. It's not technically feasible to return all records in a single api call, so HubSpot uses this paging method.

If you're looking for easier ways to paginate and query HubSpot API data, you can look at third-party API wrappers and tools, such as Kingswaysoft or CData which allows for SQL-like queries against HubSpot without fussing with the underlying API mechanics. I hope this helps!

Re: Unable to fetch more than 100 records from Hubspot API

DKumar — Mon, 19 Apr 2021 10:29:42 GMT

Hi All,

We are actually trying do an API Integration of HubSpot with our system. For this, we are trying to use a single URL for reading all the records in contact but we are not able to fetch all record in single URL for this we have to update the URL for every 100 Records due to hubspot paging. for this issue we have suggested a solution to look at third-party API wrappers and tools, such as Kingswaysoft or CData which allows for SQL-like queries against HubSpot without fussing with the underlying API mechanics. But we don't want to use third-party API wrappers and tools so except this is there any other solution through which we can fetch all contact record in singel URL.

Re: Unable to fetch more than 100 records from Hubspot API

lynton — Mon, 19 Apr 2021 13:53:28 GMT

@DKumar multiple api requests using a paging token is a requirement of any rest API, as it should not be expected that a single http request can respond with all records in the system. The response payload would be too large. If you aren't looking to use any third-party API wrappers, you will need to support the paging tokens to loop through all records from the HubSpot API.

Re: Unable to fetch more than 100 records from Hubspot API

DKumar — Tue, 01 Jun 2021 04:35:14 GMT

@lynton Thanks for your Response.Could you please explore me that how we can support paging tokens to loop through all records from the Hubspott API.

Re: Unable to fetch more than 100 records from Hubspot API

lynton — Tue, 01 Jun 2021 13:23:04 GMT

@DKumar this depends on the programming language, but any software developer should be able to apply the paging method to their code. If you're looking for off the shelf software to make it easier, check out products from Skyvia, CData, Kingswaysoft, or Stitch Data for various options to extract this data with minimal or no code.