How to paginate with cursor?

foxsterben · ‎Jun 7, 2021

When I make a GET call the response returns something like:

"paging": {
"next": {
"after": "1601",
"link": "https://api.hubapi.com/crm/v3/objects/contacts?after=1601"
}
}

however, since I have a large number (>476k) of contact when doing the hubsepot paging it returns an error. I have searched in the forum how to filter the GET call but Hubspot workers mention that to filter you must use POST which returns me something like.

"paging": {
"next": {
"after": "10"
}
}

for paging I use Azure Data Factory, as follows $.paging.next.link

But with POST I get back a cursor, and if I use $.paging.next.after I get an error.

The authentication is done with oauth2.

foxsterben · ‎Jun 15, 2021

I am already checking with technical support, and indeed the error is on their side. It should be noted that if the limit of requests is reached, the response returned would be different.

View solution in original post

GBobrowski · ‎Feb 17, 2022

I am sending various numbers appended to , like 4,7,154 , neverthe less I am getting always the same 10 records, no advancement , no change in the size of returned set.

Question 1: Is there a way to simply get all? I have only 2000 records.

Question 2: Could it be that I have a problem because I am using a demo version of HubSpot?

&limit=10&offset=10

https://api.hubapi.com/cms/v3/blogs/posts?hapikey=YOUR_HUBSPOT_API_KEY&sort=-updatedAt&&language__not_null&limit=10&offset=10 \

chandrareddy1 · ‎Aug 10, 2023

Is this resolved ? , i'm also experincing the same issue

LPM · ‎Feb 17, 2022

There seems to be an extra "&" in your query:

updatedAt&&language

I suggest removing that and try again.

foxsterben · ‎Jun 9, 2021

Yes, the "id" should not be there, I corrected it. According to the documentation, POST is used for filtering, however, the problem with this is that in the response it only gives me a cursor (i.e after=10), it does not give me a link to make the following call compared to GET

In login I have my authentication oauth2

If I try with POST, with pagintation as follows $.paging.next.after

Obviously get this error:

Operation on target Ingest_REST_Data failed: Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 404 indicates failure. Request URL: https://api.hubapi.com/90 Response payload:<!DOCTYPE html><html lang="en"></html><head><meta charset="utf-8"/><title>Error</title><link rel="icon" href="data:;base64,="><script>function refreshFunc() { window.location.reload(true);

Also I tried this

$.concat('crm/v3/objects/contacts/search?after=',$.paging.next.after)

and return this error

Operation on target Ingest_REST_Data failed: Failure happened on 'Source' side. ErrorCode=UserErrorFailToReadFromRestResource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Path ended with an open string.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Newtonsoft.Json.JsonException,Message=Path ended with an open string.,Source=Newtonsoft.Json,'

It is in the paging where the error occurs, I would appreciate your help to solve this problem.

LPM · ‎Jun 9, 2021

@foxsterben not sure what's going on with you being told you need a POST call? How did you describe your issue to support?

The POST call is for the Search API call, which has an option to filter.

https://api.hubapi.com/crm/v3/objects/contacts/search

Or the Read a batch of contacts by internal ID, or unique property values API call:

https://api.hubapi.com/crm/v3/objects/contacts/batch

If you're using the standard contact List API, then it's only GET:

https://api.hubapi.com/crm/v3/objects/contacts

What do you mean when you want to filter?

foxsterben · ‎Jun 11, 2021

@LPM

I was using POST because with GET I was getting an error indicating that the limit had been exceeded (I don't have it at the moment), there are 476k contacts, so I assumed that this was it, that's why I intended to use POST to filter the results so that they were not so many.

However, I found that actually the error using GET was due to the limits in the amount of requests per second.

https://developers.hubspot.com/docs/api/usage-details

So I limited the amount of requests so as not to exceed the limits but now I am getting this error

What happens? if it is 502 it is a problem on your side.

LPM · ‎Jun 15, 2021

@foxsterben the limit param doesn't limit the number of requests you send, it limits the number of results coming in.

If you want to control the number of requests on your server side, that's something that needs to be done on your end. It's best you check out how to throttle requests on your application and try again.

If you reach the daily alloted limit for API calls, you will need to wait till it resets until you try again.

foxsterben · ‎Jun 15, 2021

I am already checking with technical support, and indeed the error is on their side. It should be noted that if the limit of requests is reached, the response returned would be different.

LPM · ‎Jun 9, 2021

So after checking the docs and looking at the link you've posted on here, there are a few things wrong:

There shouldn't be an id parameter in the query
The after parameter should be what you get from the previous call.

i.e:

"paging": {
    "next": {
      "link": "?after=NTI1Cg%3D%3D",
      "after": "NTI1Cg%3D%3D"
    }
  }

In this example, the after value should be NTI1Cg%3D%3D so the link you use should be:

https://api.hubapi.com/crm/v3/objects/contacts?after=NTI1Cg%3D%3D

Edit:

Apologies, re-read the question and it seems you're already using what you got from the response as the call. As for filtering, can't you just limit the results using the limit param or restrict the amount of data being sent by defining the properties property?

natsumimori · ‎Jun 8, 2021

Hi @foxsterben , thank you for your post!

@LPM and @himanshurauthan , would you mind sharing your advice for @foxsterben ?

APIs & Integrations