Python KeyError When Trying to Get All Companies

[taran42]

I am trying to use Python to get all of my Company contacts. However I keep getting a KeyError for the "has-more" function. From what I've read elsewhere, I think it actually has something to do with the "response_dict" being blank, but I'm not sure how to fix it.

import requests
import json
import urllib

max_results = 10 
hapikey = 'myAPIkey' 
limit = 10 
company_list = []
get_all_companies_url = "https://api.hubapi.com/companies/v2/companies/paged?hapikey=myAPIkey"
parameter_dict = {'hapikey': hapikey, 'limit': limit}
headers = {}

# Paginate your request using offset
has_more = True
while has_more:
	parameters = urllib.urlencode(parameter_dict)
	get_url = get_all_companies_url + parameters
	r = requests.get(url= get_url, headers = headers)
	response_dict = json.loads(r.text)
	has_more = response_dict['has-more']
	company_list.extend(response_dict['companies'])
	parameter_dict['offset']= response_dict['offset']
	if len(company_list) >= max_results: # Exit pagination, based on whatever value you've set your max results variable to.
		print('maximum number of results exceeded')
		break
 
print('loop finished')

list_length = len(company_list) 

print("You've succesfully parsed through {} company records and added them to a list".format(list_length))

The error I am getting is this: 

PS C:\Users\hubspot> & "C:/Program Files/Python38/python.exe" "c:/Users/Dropbox/Hubspot/companies.py"    
Traceback (most recent call last):
  File "c:/Users/Droppbox/Hubspot/companies.py", line 30, in <module>
KeyError: 'has-more'

[taran42 · Accepted solution]

@dennisedson That is kind of what I thought as well, and I tried that, but it didn't work.

@wfong Yes, something was wrong with the dictionary. That was my thought too. Also, you are correct, I did not need my API key in the URL.

Eventually I was able to figure it out. There were a few things wrong with the while loop in regards to the response_dict call. I needed .get after the response_dict was called in three places (and a little bit of syntax fixing). See below.

import requests
import json
import urllib

max_results = 10 
hapikey = 'myAPIkey' 
limit = 10 
company_list = []
get_all_companies_url = "https://api.hubapi.com/companies/v2/companies/paged?"
parameter_dict = {'hapikey': hapikey, 'limit': limit}
headers = {}

# Paginate your request using offset
has_more = True
while has_more:
	parameters = urllib.urlencode(parameter_dict)
	get_url = get_all_companies_url + parameters
	r = requests.get(url= get_url, headers = headers)
	response_dict = json.loads(r.text)
	has_more = response_dict.get('has-more') or False
	deal_list.extend(response_dict.get('deals') or [])
	parameter_dict['Offset']= response_dict.get('offset')
    # Exit pagination, based on what ever value you've set your max results variable to
	if len(deal_list) >= max_results:
		print('maximum number of results exceeded')
		break
 
print('loop finished')

list_length = len(company_list) 

print("You've succesfully parsed through {} company records and added them to a list".format(list_length))

[wfong]

I think it is failing on the dictionary look up since there is no has-more field being returned on an error response of the get request.

I would double check the request is returning data and not an failed response.

I see the line:

get_all_companies_url = "https://api.hubapi.com/companies/v2/companies/paged?hapikey=myAPIkey"

but shouldn't it be only

get_all_companies_url = "https://api.hubapi.com/companies/v2/companies/paged?"

You already have this part of the code which contains the hapikey.

parameter_dict = {'hapikey': hapikey, 'limit': limit}

That's my guess!

[dennisedson]

Hello @taran42 !

Long time no chat.  As you know, I am not expert at Python, but I am wondering if you can add an if statement within that loop basically saying if has-more is null, then has-more = false...

@akaiser , @khookguy , @wfong !  Hello my Python friends!  Do you guys have thoughts on this?  I also don't mind if you all look at my answer and start a private chat to laugh at me 😅  for suggesting something so outrageous!  

Best!