Contacts Search API not returning all records

[mfed_303]

Hello, I have the following python code block. Unfortunately, it only returns 1000 out of 78K possible results. Not sure what I’m missing since I’m taking into account both pagination and rate limit of 100 calls / 10 seconds. Any help is appreciated.

import requestsimport jsonimport pandas as pdimport timedef flatten_json(y):out = {}def flatten(x, name=''):if type(x) is dict:for a in x:flatten(x[a], name + a + '_')elif type(x) is list:i = 0for a in x:flatten(a, name + str(i) + '_')i += 1else:out[name[:-1]] = xflatten(y)return out# Bearer token for authenticationbearer_token = "pretend_token"# Base URL for searching contactsbase_url = "https://api.hubapi.com/crm/v3/objects/contacts/search"# Headers with Bearer token for authenticationheaders = {'Authorization': f'Bearer {bearer_token}','Content-Type': 'application/json'}# Initialize variables for pagination and rate limitingall_contacts = []after = 0limit = 100has_more = Truestart_time = time.time()request_count = 0MAX_REQUESTS = 15 # Max requests per windowWINDOW_SIZE = 10 # Window size in secondswhile has_more:current_time = time.time()elapsed_time = current_time - start_time# Check for rate limitingif elapsed_time > WINDOW_SIZE:start_time = current_timerequest_count = 0elif request_count >= MAX_REQUESTS:time.sleep(WINDOW_SIZE - elapsed_time)start_time = time.time()request_count = 0# Request body with filters and paginationpayload = json.dumps({"filterGroups": [{"filters": [{"propertyName": "example_prop name","operator": "EQ","value": "true"}]}],"limit": limit,"after": after})try:# POST request to search contacts with paginationresponse = requests.request("POST", base_url, headers=headers, data=payload)data = response.json()# Check if 'results' key is in responseif 'results' in data:# Flatten and extend the resultsflat_data = [flatten_json(item) for item in data['results']]all_contacts.extend(flat_data)# Update 'after' for next paginationif 'paging' in data and 'next' in data['paging']:after = data['paging']['next']['after']else:has_more = Falseelse:print("No 'results' key found in the response.")breakexcept requests.exceptions.RequestException as e:print(f"Request failed: {e}")breakrequest_count += 1# Convert all flattened results to a DataFramecontacts_df = pd.DataFrame(all_contacts)# Print the DataFrameprint(contacts_df)

[Jaycee_Lewis]

Hi, @mfed_303 Welcome to our community.

Were you able to make any progress with your troubleshooting? If not, I have a few questions to help give our community members more context.

It seems like there might be a few areas in your code that could be causing the issue of not retrieving all the desired records:

1. Rate-Limiting Logic: Have you verified that your rate-limiting logic is functioning as expected? It’s important to ensure that your script is correctly handling the API’s rate limits
2. Debugging the after Variable: Could you add a debugging step to check if the after variable is being updated correctly after each API call? This is crucial for proper pagination and fetching all records
3. Simplify Your Request: Have you tried simplifying your request to remove the complexity of the nested if/then statements? Sometimes, simplifying the code can help identify where the issue might be

By checking these aspects, you might be able to pinpoint the cause of the issue and ensure that your script fetches all the contacts as intended.

Please let us know how it goes! — Jaycee

[mfed_303]

Hello Jaycee

I simplified the code for the rate limit, essentially making sure that only 1 call is made every second. I also printed the values for request number, after, list length, and datetime after each call. And they seem correct to me (see picture). Not sure what I’m doing wrong.

Here is my revised code block

while has_more:

# Request body with filters and pagination

payload = json.dumps({

“filterGroups”: [{

“filters”: [{

“propertyName”: “name_text”,

“operator”: “EQ”,

“value”: “true”

}]

}],

“limit”: limit,

“after”: after

})

try:

# POST request to search contacts with pagination

response = requests.request(“POST”, base_url, headers=headers, data=payload)

data = response.json()

# Check if ‘results’ key is in response

if ‘results’ in data:

# Flatten and extend the results

flat_data = [flatten_json(item) for item in data[‘results’]]

all_contacts.extend(flat_data)

# Update ‘after’ for next pagination

if ‘paging’ in data and ‘next’ in data[‘paging’]:

after = data[‘paging’][‘next’][‘after’]

else:

has_more = False

else:

print(“No ‘results’ key found in the response.”)

break

except requests.exceptions.RequestException as e:

print(f"Request failed: {e}")

break

list_length = len(all_contacts)

request_count += 1

current_time = datetime.datetime.now()

print('Request Count: ', request_count)

print('After Value: ', after)

print('List Length: ', list_length)

print('Current Time: ', current_time)

# Sleep for one second to ensure only one API call per second

time.sleep(1)

# Convert all flattened results to a DataFrame

contacts_df = pd.DataFrame(all_contacts)

# Print the DataFrame

print(contacts_df)