Get All Deals using API v3 and Pagination

[SSekela]

Hi friends.

I'm new to the Hubspot API's (and API processing in general).  I was able to access a page of Deals using the version (v3) API.  I successfully got a page of deals (ie 100) using BOTH of the following calls

hubspotClient.crm.deals.basicApi.getPage

AND

https://api.hubapi.com/crm/v3/objects/deals.

I've tested this process in BOTH node.js AND Google App Scripts.  My question is this: Can someone point me to a simple example of how to paginate through the Deal object so I can process ALL the deals? (not just 100)?  I've seen examples of how to do this, but they seem to all point to using the older v1 API methods.  I want to be consistent and use the latest API calls.  (v3)

Thanks for your help!

Shawn

[SSekela · Accepted solution]

Dear Dennis,

Thanks for your response!  Yes, I actually did what you said in your first suggestion (using the stack overflow post and replacing version v1 with v3 constructs.  My code doesn't follow the recursive function logic from the stack overflow sample - I used an older "Do While" approach and set a flag to stop looping depending on whether the API returned a "next page".  Again - I'm an "older" developer from the 1980s 90, and 2000s.  Mostly traditional back-end Database,  top-down structured programming, and development.  The newer API and Web tools, REST API are very new to me - but I'm learning pretty quickly.  Finally - thanks for pointing out the other two (nodejs) references.  I've seen one of them before, but I did not see that special git-hub project for hubspot objects.  That looks very interesting.  Learning a lot about Google Apps Scripts and node.js these days.  Cool stuff.  Sorry for rambling.  Us tech people like to blab sometimes about boring stuff!  Take care!

Shawn

[RobCuellari · Accepted solution]

I wrote a pretty simple while loop to get all the deals via the python client (import hubspot)

(note I imported the time library to time.sleep(0.5) so I hopefully account for any rate limit.

deals = []

next_page = 1
while True:
    time.sleep(0.5)
    if next_page == 1:
        api_response = client.crm.deals.basic_api.get_page(limit=100, archived=False)
        for item in api_response.results:
            deals.append(item)
            next_page = api_response.paging.next.after
    else:
        api_response = client.crm.deals.basic_api.get_page(limit=100, archived=False, after=next_page)
        if len(api_response.results) > 0:
            for item in api_response.results:
                deals.append(item)
            try:
                next_page = api_response.paging.next.after
                api_response.paging.next.link
            except AttributeError:
                break

[KMalaiyappan]

url = 'https://api.hubapi.com/crm/v3/objects/deals'
headers = {'authorization':('Bearer '+ private_app_key)}
params = {'archived': 'false', 'limit': 100} # Adjust the parameters as needed

results = []
response = requests.get(url, headers=headers, params=params)
data = response.json()

while True:
results.extend(data['results'])
try:
url = data['paging']['next']['link']
except Exception as e:
print('The Error is: ', e)
break
response = requests.get(url, headers=headers)

#print(response.status_code,response.reason)
data = response.json()

[RobCuellari · Accepted solution]

I wrote a pretty simple while loop to get all the deals via the python client (import hubspot)

(note I imported the time library to time.sleep(0.5) so I hopefully account for any rate limit.

deals = []

next_page = 1
while True:
    time.sleep(0.5)
    if next_page == 1:
        api_response = client.crm.deals.basic_api.get_page(limit=100, archived=False)
        for item in api_response.results:
            deals.append(item)
            next_page = api_response.paging.next.after
    else:
        api_response = client.crm.deals.basic_api.get_page(limit=100, archived=False, after=next_page)
        if len(api_response.results) > 0:
            for item in api_response.results:
                deals.append(item)
            try:
                next_page = api_response.paging.next.after
                api_response.paging.next.link
            except AttributeError:
                break

[DataEngineer · Accepted solution]

Thanks a lot for this solution Rob. You inspired me to come up with a similar solution, using a while loop and pagination. Here it is:

from hubspot import HubSpot
api_client = HubSpot(access_token='your_acces_token')

deals = []
page = None  # Start with None page cursor

while True:
    print(f'Fetching page cursor: {page}')

    api_response = api_client.crm.deals.basic_api.get_page(limit=100, archived=False, after=page)
    deals.extend(api_response.results)
    
    try:
        page = api_response.paging.next.after
    except AttributeError:
        print("No next page found, pagination completed.")
        break

This simple solution seems to work for me. Rate limits have not been an issue so far. 

[dennisedson]

@SSekela 

I believe I saw you on another thread where I posted a link to a stack overflow thread that had the answer.  In the code, they are using the older v1 when the varial url is set.  You can replace that with the v3 version as well as update any other parameters you would like to be returned. 

Another place to look is this pagination tool on GitHub

And while we are on GitHub, there is this node SDK that you could explore.

[SSekela · Accepted solution]

Dear Dennis,

Thanks for your response!  Yes, I actually did what you said in your first suggestion (using the stack overflow post and replacing version v1 with v3 constructs.  My code doesn't follow the recursive function logic from the stack overflow sample - I used an older "Do While" approach and set a flag to stop looping depending on whether the API returned a "next page".  Again - I'm an "older" developer from the 1980s 90, and 2000s.  Mostly traditional back-end Database,  top-down structured programming, and development.  The newer API and Web tools, REST API are very new to me - but I'm learning pretty quickly.  Finally - thanks for pointing out the other two (nodejs) references.  I've seen one of them before, but I did not see that special git-hub project for hubspot objects.  That looks very interesting.  Learning a lot about Google Apps Scripts and node.js these days.  Cool stuff.  Sorry for rambling.  Us tech people like to blab sometimes about boring stuff!  Take care!

Shawn