topic API pagination consistency in APIs & Integrations

API pagination consistency

RLehrhaupt — Tue, 02 May 2023 19:28:19 GMT

I am making a script to loop through pythonic http requests on an object and using pagination to retrieve all objects. I am having no issue retrieving the paging data and "after" property to include in query params for the next request. However, this process is slow.

One approach I am taking is caching the "afters" so i can do asynchronous requests/parallelization. And then at the end check if there is an additional page. Then periodically update the cache.

While this works, I have some questions. Are the "after" values consistent if I change something like the amount of properties requested for an object.

So far I have tested some things. Retreiving contact pages using just 1 requested property versus 100. It seems to return the same "after" sequence but I just want to confirm.

Changing "limit" however changes the "after" pagination values. Is "limit" the only variable that can change them? Is there a better approach to use?

Re: API pagination consistency

Jaycee_Lewis — Wed, 03 May 2023 21:51:47 GMT

Hi, @RLehrhaupt 👋 Thanks for the interesting question. My gut instinct is that you are correct, changing the limit is how we can affect the “after” value. And this is a case of not having a specific document or call-out in a document on what/when/how the “after” value can be altered.

I'll do some research on my end to see if we can get clarity on this. Or get an update to the documentation for clarity purposes (not a fast solution, but it helps all of us, and I'm happy to pull on that thread)

For your question on better/other approaches, I'll invite some community members to offer their experience — hey @tominal @ChrisoKlepke, do you have any tips? Or suggestions on other ways @RLehrhaupt might tackle their project?

Best,

Jaycee

API pagination consistency

RLehrhaupt — Thu, 04 May 2023 15:20:31 GMT

Thanks for looking into it. Looks like the "after" parameter corresponds to the last returns hs_object_id+1.

Knowing this information, a somewhat tricky yet viable approach could be iteration on closed bounds. For example using these steps

Find lower bound

Request the object with max limit (limit 100) and keep track of the hs_object_id ranges found. For example maybe these were the returned in a list count of 100

100
101
10002
. (etc... imagine long list)
30024

So my lower bound is 30024 since the first page query returned that as the last item. I now have found all available IDs for range 0-30024.

Find upper bound.

Now I have to find the largest value of the hs_object_id for the object. I can send out a bunch of "after" parameters using an iterative approach until I find adjacent "after" tests where one returns info and one does not. Then I know the last ID is in that range. I can then narrow into that range and use another parallel based request to further narrow the range until I find a request that has no "after" pagination info. I can then take that last hs_object_id as my upper bound.

Iterate to fill gaps

So now I have an lower and upper bound for possible hs_object_ids for my object. I can then parse intervals and send parallel requests to find objects within ranges, mark ranges as completed (in terms of search). This allows me to send out parallel requests to find all hs_object_ids that exist for my object.

Assemble "after" list for desired "limit"

Now that I have a list of all hs_object_ids, I can split up the list into groups of 100 (assuming I use a 'limit' parameter of 100 later for object requests). Now I have subgroups of 100 and can use the first hs_object_id in each group as my "afters" to go through parallel pagination request for my objects later. Note that even though the typical "afters" use hs_object_id+1 using the last item of the page, using the first hs_object_id of each group will provide the same results, it's really a preference. This method will assemble a list of existing hs_object_ids whereas hs_object_id+1 for last items does not guarantee the "after" value is an existing hs_object_id.

This is all just hypothetical as of now, I plan to test this. Sadly, this is alot of work, I wish there was an endpoint for objects just to return a list of hs_object_ids so formulating the pagination parameters was simpler. The return value could be formated in compressed for such that if hs_object_ids 101,102,103,105,108 exist (for example) it would return 101-105, 108. This would reduce the possibility of needing pagination for this hypothetical endpoint itself.

Any thoughts?

Re: API pagination consistency

tominal — Thu, 04 May 2023 19:47:05 GMT

Hey @RLehrhaupt,

That is kind of where my head is going. You could perform two search endpoint requests that get your lower and upper bounds, then split up the requests to run on separate threads in parallel.

The problem is I do not know what happens if you delete a record, but still use its ID for an "after" parameter.

Hope that helps!

Re: API pagination consistency

RLehrhaupt — Thu, 04 May 2023 20:48:24 GMT

After some testing, using "after" parameters for hs_object_ids that do not exist do not appear to break anything, the page results will just start at the first existing hs_object_id above or equal to the parameter. So far I have been able to efficiently capture closed bounds for an object using an iterative approach.

Since hs_object_id's tend to have sparse groupings within the closed bound range the tricky part will be avoiding querying unpopulated regions along the range/bound. Based on my test capturing the upper bound I think this approach will work.

My plan is to cache the "after" parameters every day or so which will allow me to use async for paging through objects.