topic Re: Offset makes static lists API v1 endpoint unusable (can't paginate records for enterprise custom in APIs & Integrations

Offset makes static lists API v1 endpoint unusable (can't paginate records for enterprise customer)

RichSteinmetz — Fri, 09 Jun 2023 07:15:32 GMT

Hey folks,

We have a technical issue that currently affects 1 customer and it's not apparent from the API responses how to handle it (it might be an API bug).

A customer has more than 250 static lists. We try to paginate but the behavior of the offset is very strange.

With 3 records the endpoint always returns an offset of 12. Or 14 if we don't provide an offset. Or 1 if the offset is very high. Here are some example requests that we tried:

irb(main):025:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=1").parse(:json)
=>
{"offset"=>12,
 "total"=>1,
 "lists"=>
  [{"portalId"=>21303762,
    "listId"=>2,
    "createdAt"=>1649406615866,
    "updatedAt"=>1649406615932,
    "name"=>"ali-contact-list-static",
    "listType"=>"STATIC",
    "authorId"=>3969928,
    "filters"=>[],
    "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
    "archived"=>false,
    "teamIds"=>[],
    "ilsFilterBranch"=>"",
    "readOnly"=>false,
    "internal"=>false,
    "limitExempt"=>false,
    "dynamic"=>false}],
 "has-more"=>true}

irb(main):026:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=2").parse(:json)
=>
{"offset"=>12,
 "total"=>1,
 "lists"=>
  [{"portalId"=>21303762,
    "listId"=>2,
    "createdAt"=>1649406615866,
    "updatedAt"=>1649406615932,
    "name"=>"ali-contact-list-static",
    "listType"=>"STATIC",
    "authorId"=>3969928,
    "filters"=>[],
    "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
    "archived"=>false,
    "teamIds"=>[],
    "ilsFilterBranch"=>"",
    "readOnly"=>false,
    "dynamic"=>false,
    "limitExempt"=>false,
    "internal"=>false}],
 "has-more"=>true}

irb(main):027:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=0").parse(:json)
=>
{"offset"=>12,
 "total"=>1,
 "lists"=>
  [{"portalId"=>21303762,
    "listId"=>2,
    "createdAt"=>1649406615866,
    "updatedAt"=>1649406615932,
    "name"=>"ali-contact-list-static",
    "listType"=>"STATIC",
    "authorId"=>3969928,
    "filters"=>[],
    "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
    "archived"=>false,
    "teamIds"=>[],
    "ilsFilterBranch"=>"",
    "readOnly"=>false,
    "internal"=>false,
    "limitExempt"=>false,
    "dynamic"=>false}],
 "has-more"=>true}

irb(main):028:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=100").parse(:json)
=>
{"offset"=>1, "total"=>0, "lists"=>[], "has-more"=>false}

irb(main):029:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static").parse(:json)
=>
{"offset"=>14,
 "total"=>3,
 "lists"=>
  [{"portalId"=>21303762,
    "listId"=>2,
    "createdAt"=>1649406615866,
    "updatedAt"=>1649406615932,
    "name"=>"ali-contact-list-static",
    "listType"=>"STATIC",
    "authorId"=>3969928,
    "filters"=>[],
    "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
    "archived"=>false,
    "teamIds"=>[],
    "ilsFilterBranch"=>"",
    "readOnly"=>false,
    "dynamic"=>false,
    "limitExempt"=>false,
    "internal"=>false},
   {"portalId"=>21303762,
    "listId"=>3,
    "createdAt"=>1686215954821,
    "updatedAt"=>1686215955134,
    "name"=>"ali-contact-list-static (cloned)",
    "listType"=>"STATIC",
    "authorId"=>45561614,
    "filters"=>[],
    "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1686215955434, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1686215955216, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
    "archived"=>false,
    "teamIds"=>[],
    "ilsFilterBranch"=>"",
    "readOnly"=>false,
    "dynamic"=>false,
    "limitExempt"=>false,
    "internal"=>false},
   {"portalId"=>21303762,
    "listId"=>4,
    "createdAt"=>1686215989072,
    "updatedAt"=>1686215989357,
    "name"=>"ali-contact-list-static (cloned) 2",
    "listType"=>"STATIC",
    "authorId"=>45561614,
    "filters"=>[],
    "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1686215989857, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1686215989473, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
    "archived"=>false,
    "teamIds"=>[],
    "ilsFilterBranch"=>"",
    "readOnly"=>false,
    "dynamic"=>false,
    "limitExempt"=>false,
    "internal"=>false}],
 "has-more"=>false}

Re: Offset makes static lists API v1 endpoint unusable (can't paginate records for enterprise custom

Jaycee_Lewis — Fri, 09 Jun 2023 22:39:53 GMT

Hey, @RichSteinmetz 👋 Welcome to our community!

I have a few initial questions to help get us started:

For your initial requests, are you sending a `count` and `offset`? Or just an offset?
What's the result and returned `offset` when you make use an initial request URL like this?
https://api.hubapi.com/contacts/v1/lists/static?count=250Do you get an offset returned?
For this portal specifically, have you observed this issue with other endpoints or only with the static lists endpoint?
Could you please confirm the language/library you are using to make these API calls? It looks like Ruby from the code snippets you provided, but I want to be certain 😊

Thank you very much for the additional information! — Jaycee

Re: Offset makes static lists API v1 endpoint unusable (can't paginate records for enterprise custom

RichSteinmetz — Mon, 12 Jun 2023 07:10:17 GMT

Hey @Jaycee_Lewis, thanks for the welcome and for checking it :))

For your initial requests, are you sending a `count` and `offset`? Or just an offset?

Each code block examples starts with a piece of code that shows the URL that we are hitting, in both cases we are passing both.

What's the result and returned `offset` when you make use an initial request URL like this?

It's the same as not passing a count at all. The offset is 14 which to me seems pretty random given that in this dummy account we only have 3 static lists:

>> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=250").parse(:json) => {"offset"=>14, "total"=>3, "lists"=> [{"portalId"=>21303762, "listId"=>2, "createdAt"=>1649406615866, "updatedAt"=>1649406615932, "name"=>"ali-contact-list-static", "listType"=>"STATIC", "authorId"=>3969928, "filters"=>[], "metaData"=> {"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil}, "archived"=>false, "teamIds"=>[], "ilsFilterBranch"=>"", "readOnly"=>false, "limitExempt"=>false, "internal"=>false, "dynamic"=>false}, {"portalId"=>21303762, "listId"=>3, "createdAt"=>1686215954821, "updatedAt"=>1686215955134, "name"=>"ali-contact-list-static (cloned)", "listType"=>"STATIC", "authorId"=>45561614, "filters"=>[], "metaData"=> {"size"=>1, "lastSizeChangeAt"=>1686215955434, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1686215955216, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil}, "archived"=>false, "teamIds"=>[], "ilsFilterBranch"=>"", "readOnly"=>false, "limitExempt"=>false, "internal"=>false, "dynamic"=>false}, {"portalId"=>21303762, "listId"=>4, "createdAt"=>1686215989072, "updatedAt"=>1686215989357, "name"=>"ali-contact-list-static (cloned) 2", "listType"=>"STATIC", "authorId"=>45561614, "filters"=>[], "metaData"=> {"size"=>1, "lastSizeChangeAt"=>1686215989857, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1686215989473, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil}, "archived"=>false, "teamIds"=>[], "ilsFilterBranch"=>"", "readOnly"=>false, "limitExempt"=>false, "internal"=>false, "dynamic"=>false}], "has-more"=>false}

For this portal specifically, have you observed this issue with other endpoints or only with the static lists endpoint?

So far, we haven't noticed similar issues elsewhere, but we are also not paginating results anywhere else yet, I think.

Could you please confirm the language/library you are using to make these API calls? It looks like Ruby from the code snippets you provided, but I want to be certain 😊

It is Ruby indeed with our self-rolled client 🙂 (no additional libraries/gems used)

Thanks again and see you around! 👋