Get all contacts doesn't get all contacts

[#1]

Hello. I am developing a site for a club which have several franchises called "chapters" around the world. We have our members stored as contacts in Hubspot, who all have a property called "choose_chapter", which represents what chapter that they are in.

I need to write a script that figures out how many members each chapter has. It does this by scrolling through all members and counting how many are in each chapter. The only problem is that no matter what I do, I will get a lesser number compared to what I see in Hubspot's webapp. An example would be that I get 9 member in a specific chapter when I expect 22.

Here is my code (in python)
import requests
import requests
import json

dict = {}

def get_new(vid_offset):
	url = 'https://api.hubapi.com/contacts/v1/lists/180/contacts/all?hapikey=[HERE-IS-MY-API-KEY]&count=100&property=choose_chapter&vidOffset='+str(vid_offset)
	response = requests.get(url)
	json = response.json()
	for i in json["contacts"]:
		try:
			chapter = i["properties"]['choose_chapter']['value']
			try:
				formercount = dict[chapter]
			except KeyError:
				dict[chapter] = 0
			dict[chapter] += 1
		except KeyError:
			pass
	if (response.json()["vid-offset"] > vid_offset):
		vid_offset = response.json()["vid-offset"]
		get_new(vid_offset)

get_new(0)
print(dict)

Thanks!

[cbarley]

Ah! Don't know how that slipped by me. I didn't notice it in your URL at first. Glad you were able to get it solved!

[cbarley · Accepted solution]

Hi @Rasmus_Hag_Lovstad, Apologies for the delay, this one slipped through the cracks. Taking a look at the vid offsets and using the new one returned to me in each subsequent request, I was able to page through all of my contacts manually in my own HubSpot Account. I found that the same number of contacts in the UI with a certain value were also being returned back to me correctly through the API. There doesn't appear to be an issue with the vidOffset, but it's likely something to do with the implementation aspect.

I wish I were able to speak specifically to python implementation, but I'm more familiar with NodeJS, and was able to create a program that will run through a portal, check a custom property, then push those items to an array which then gets sent to a logger. I tested this tool on your portal, and was able to get 16 results for the choose_chapter property being equal to "Cambridge". I also was returned 51 contacts for "Berlin", both of which are correct values.

If you're curious, here's my repo: https://github.com/cbarley10/hubspot-pagination/blob/master/app.js

One thing in terms of your code that you may want to watch out for is that vids are random and non-sequential. It looks like you may be comparing vids to say if if the response's vid offset is greater than the current offset, then to get a new value, but you should really be checking against the has-more property being true or false. If has-more is true, you should make another GET request to the endpoint with the returned vidOffset. If has-more is false, you're done paging and should return all of your data.

[#4]

Hello, just wanted to give an update:

I solved the problem by changing the list number. I used to use this url:
https://api.hubapi.com/contacts/v1/lists/180/contacts/all?hapikey=[HERE-IS-MY-API-KEY]&count=100&pro...

Instead I now use following url:
https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=[HERE-IS-MY-API-KEY]&count=100&pro...

I changed the '180' to 'all'

This solved my problem.

[cbarley]

Hi @Rasmus_Hag_Lovstad, I don't see any reason why the endpoint would not return the correct number of contacts with that property. I'm going to continue to do some testing and see if there's anything that might be wrong with the vid-offset/ has-more fields. Thanks for your patience!

[cbarley]

Hi @Rasmus_Hag_Lovstad, happy to help here - can you reply with the portal ID you're trying to get this info from?

[#7]

Not sure if this is what you're looking for, but 596010