Offset makes static lists API v1 endpoint unusable (can't paginate records for enterprise customer)

RichSteinmetz · June 9, 2023, 7:13am

Hey folks,

We have a technical issue that currently affects 1 customer and it’s not apparent from the API responses how to handle it (it might be an API bug).

A customer has more than 250 static lists. We try to paginate but the behavior of the offset is very strange.

With 3 records the endpoint always returns an offset of 12. Or 14 if we don’t provide an offset. Or 1 if the offset is very high. Here are some example requests that we tried:

irb(main):025:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=1").parse(:json)
=>
{"offset"=>12,
 "total"=>1,
 "lists"=>
 [{"portalId"=>21303762,
 "listId"=>2,
 "createdAt"=>1649406615866,
 "updatedAt"=>1649406615932,
 "name"=>"ali-contact-list-static",
 "listType"=>"STATIC",
 "authorId"=>3969928,
 "filters"=>[],
 "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "internal"=>false,
 "limitExempt"=>false,
 "dynamic"=>false}],
 "has-more"=>true}

irb(main):026:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=2").parse(:json)
=>
{"offset"=>12,
 "total"=>1,
 "lists"=>
 [{"portalId"=>21303762,
 "listId"=>2,
 "createdAt"=>1649406615866,
 "updatedAt"=>1649406615932,
 "name"=>"ali-contact-list-static",
 "listType"=>"STATIC",
 "authorId"=>3969928,
 "filters"=>[],
 "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "dynamic"=>false,
 "limitExempt"=>false,
 "internal"=>false}],
 "has-more"=>true}

irb(main):027:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=0").parse(:json)
=>
{"offset"=>12,
 "total"=>1,
 "lists"=>
 [{"portalId"=>21303762,
 "listId"=>2,
 "createdAt"=>1649406615866,
 "updatedAt"=>1649406615932,
 "name"=>"ali-contact-list-static",
 "listType"=>"STATIC",
 "authorId"=>3969928,
 "filters"=>[],
 "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "internal"=>false,
 "limitExempt"=>false,
 "dynamic"=>false}],
 "has-more"=>true}

irb(main):028:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=1&offset=100").parse(:json)
=>
{"offset"=>1, "total"=>0, "lists"=>[], "has-more"=>false}

irb(main):029:0> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static").parse(:json)
=>
{"offset"=>14,
 "total"=>3,
 "lists"=>
 [{"portalId"=>21303762,
 "listId"=>2,
 "createdAt"=>1649406615866,
 "updatedAt"=>1649406615932,
 "name"=>"ali-contact-list-static",
 "listType"=>"STATIC",
 "authorId"=>3969928,
 "filters"=>[],
 "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1649426380447, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1649406616513, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "dynamic"=>false,
 "limitExempt"=>false,
 "internal"=>false},
 {"portalId"=>21303762,
 "listId"=>3,
 "createdAt"=>1686215954821,
 "updatedAt"=>1686215955134,
 "name"=>"ali-contact-list-static (cloned)",
 "listType"=>"STATIC",
 "authorId"=>45561614,
 "filters"=>[],
 "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1686215955434, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1686215955216, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "dynamic"=>false,
 "limitExempt"=>false,
 "internal"=>false},
 {"portalId"=>21303762,
 "listId"=>4,
 "createdAt"=>1686215989072,
 "updatedAt"=>1686215989357,
 "name"=>"ali-contact-list-static (cloned) 2",
 "listType"=>"STATIC",
 "authorId"=>45561614,
 "filters"=>[],
 "metaData"=>{"size"=>1, "lastSizeChangeAt"=>1686215989857, "processing"=>"DONE", "lastProcessingStateChangeAt"=>1686215989473, "error"=>"", "listReferencesCount"=>nil, "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "dynamic"=>false,
 "limitExempt"=>false,
 "internal"=>false}],
 "has-more"=>false}

Jaycee_Lewis · June 9, 2023, 10:39pm

Hey, @RichSteinmetz Welcome to our community!

I have a few initial questions to help get us started:

For your initial requests, are you sending a `count` and `offset`? Or just an offset?
What’s the result and returned `offset` when you make use an initial request URL like this?
https://api.hubapi.com/contacts/v1/lists/static?count=250Do you get an offset returned?
For this portal specifically, have you observed this issue with other endpoints or only with the static lists endpoint?
Could you please confirm the language/library you are using to make these API calls? It looks like Ruby from the code snippets you provided, but I want to be certain

Thank you very much for the additional information! — Jaycee

RichSteinmetz · June 12, 2023, 7:10am

Hey @Jaycee_Lewis, thanks for the welcome and for checking it :))

For your initial requests, are you sending a `count` and `offset`? Or just an offset?

Each code block examples starts with a piece of code that shows the URL that we are hitting, in both cases we are passing both.

What’s the result and returned `offset` when you make use an initial request URL like this?

It’s the same as not passing a count at all. The offset is 14 which to me seems pretty random given that in this dummy account we only have 3 static lists:

>> Integration::Hubspot.last.service.client.request.get("https://api.hubapi.com/contacts/v1/lists/static?count=250").parse(:json)

=> 
{"offset"=>14,
 "total"=>3,
 "lists"=>
 [{"portalId"=>21303762,
 "listId"=>2,
 "createdAt"=>1649406615866,
 "updatedAt"=>1649406615932,
 "name"=>"ali-contact-list-static",
 "listType"=>"STATIC",
 "authorId"=>3969928,
 "filters"=>[],
 "metaData"=>
 {"size"=>1,
 "lastSizeChangeAt"=>1649426380447,
 "processing"=>"DONE",
 "lastProcessingStateChangeAt"=>1649406616513,
 "error"=>"",
 "listReferencesCount"=>nil,
 "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "limitExempt"=>false,
 "internal"=>false,
 "dynamic"=>false},
 {"portalId"=>21303762,
 "listId"=>3,
 "createdAt"=>1686215954821,
 "updatedAt"=>1686215955134,
 "name"=>"ali-contact-list-static (cloned)",
 "listType"=>"STATIC",
 "authorId"=>45561614,
 "filters"=>[],
 "metaData"=>
 {"size"=>1,
 "lastSizeChangeAt"=>1686215955434,
 "processing"=>"DONE",
 "lastProcessingStateChangeAt"=>1686215955216,
 "error"=>"",
 "listReferencesCount"=>nil,
 "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "limitExempt"=>false,
 "internal"=>false,
 "dynamic"=>false},
 {"portalId"=>21303762,
 "listId"=>4,
 "createdAt"=>1686215989072,
 "updatedAt"=>1686215989357,
 "name"=>"ali-contact-list-static (cloned) 2",
 "listType"=>"STATIC",
 "authorId"=>45561614,
 "filters"=>[],
 "metaData"=>
 {"size"=>1,
 "lastSizeChangeAt"=>1686215989857,
 "processing"=>"DONE",
 "lastProcessingStateChangeAt"=>1686215989473,
 "error"=>"",
 "listReferencesCount"=>nil,
 "parentFolderId"=>nil},
 "archived"=>false,
 "teamIds"=>[],
 "ilsFilterBranch"=>"",
 "readOnly"=>false,
 "limitExempt"=>false,
 "internal"=>false,
 "dynamic"=>false}],
 "has-more"=>false}

For this portal specifically, have you observed this issue with other endpoints or only with the static lists endpoint?

So far, we haven’t noticed similar issues elsewhere, but we are also not paginating results anywhere else yet, I think.

Could you please confirm the language/library you are using to make these API calls? It looks like Ruby from the code snippets you provided, but I want to be certain

It is Ruby indeed with our self-rolled client (no additional libraries/gems used)

Thanks again and see you around!

Topic		Replies	Views
Contact Lists Pagination APIs & Integrations professional	2	1802	February 14, 2023
Export data automatically under then constrain on offset APIs & Integrations contacts	1	903	November 12, 2018
contacts/v1/lists Pagination doesn't work APIs & Integrations	6	4086	July 15, 2022
Get all companies - odd offset behaviour APIs & Integrations companies	5	5438	October 3, 2020
Get recently updated contacts pagination issue APIs & Integrations contacts	3	1149	May 13, 2021

Offset makes static lists API v1 endpoint unusable (can't paginate records for enterprise customer)

Related topics