APIs & Integrations

juanluisrivero
Member

Incremental upload

SOLVE

Hello, I'm new with HubSpot.

 

I am working with incremental loads of Contacts and Deals. For Contacts incremental loads I do the following:

  1. I ask for all the contacts and I'm left with the last timestamp that appears in "addedAt".
  2. For the successive incremental loads I keep the New Contacts created from that last timestamp.
  3. For the successive loads of Updates I also have the contacts that have changed since that timestamp.

Am I doing it right? I do not have any data that is new or modified and I have not noticed?

 

On the other hand, I want to do the same with "Deals". Deals has a parameter for New and Updated Deals : "since". You must set 'since' = timestamp. The problem arises from that I do not know what "timestamp" I should catch. Can anyone help me with this timestamp that I have to take into account to take the Deals from there onwards?

 

Thank you very much in advance.

 

Regards.

0 Upvotes
1 Accepted solution
IsaacTakushi
Solution
HubSpot Employee
HubSpot Employee

Incremental upload

SOLVE

Hi, @juanluisrivero.

 

Thank you for clarifying; I believe I do have a better idea of your goal.

 

It sounds like you are retrieving contacts and deals from HubSpot, adding them to an external database, and then trying to keep the external records updated with periodic uploads.

 

If you are seeking a date property to indicate when the new and modified deals begin, I recommend using Last modified date (hs_lastmodifieddate), which records the last time any deal property was updated. This includes deal creations.

 

That said, I see a few other alternatives to your incremental upload approach. (Although I talk about deals below, the same concepts apply to contacts.):

  • Once you have uploaded the deals, you could periodically query the Get recently modified deals endpoint and page backward until you find a timestamp value you have already recorded. This approach conserves more calls than using the Get all deals endpoint alone.
  • If you create an app and connect it to your HubSpot account with OAuth, your app can notify you when contacts and deals are created or change via the Webhooks API. The Webooks API allows you to create "subscriptions" to events in accounts where your app is installed. When the chosen events occur, the application will send your service a notification with the details of the contact or deal which was created or changed. This way, you don't have to poll HubSpot's Get all endpoints for updated records.

Isaac Takushi

Associate Certification Manager

View solution in original post

0 Upvotes
4 Replies 4
IsaacTakushi
HubSpot Employee
HubSpot Employee

Incremental upload

SOLVE

Welcome, @juanluisrivero!

 

Could you clarify your goal for these "incremental loads?" Are you creating new contacts in HubSpot and updating existing contacts at the same time?

 

If so, you can do this without using any timestamp properties. HubSpot will automatically:

  • Create new contacts if they do not exist.
  • Update existing contacts with any new information.
  • Stamp the time of update on all new and existing contacts.

Am I misunderstanding your use of the timestamps? For example, do you use the timestamps to sync with some external database?

 

Yes, the Get recently modified deals endpoint accepts a since parameter. For the "next" timestamp, people usually have their system record the UNIX time they last called the endpoint so they can use it on the next call, or use the most recent timestamp property value. Since the endpoint returns results starting with the most recently modified deals, this value will be in the very first deal returned.

Isaac Takushi

Associate Certification Manager
0 Upvotes
juanluisrivero
Member

Incremental upload

SOLVE

Hello Isaac,

I have to make an ingestion of Contacts and Deals mainly via REST API. I have 830.000 deals and 640.000 contacts. I have to request them 3.000 times for Deals and 6.400 for Contacts. These processes spend a lot of time to request "all" each time you want to get the info.

For this reason, I have planned to make incremental upload:

1) First I get "all" contacts and deals.

2) Next download I only get the new ones.

For Contacts it is easy, if you get "all" the first time and calculate "lastAddedAt", as the new contacts are sorted, you stop when the lastAddedAt is greater than the lastAddedAt you have in your big contact request (calculated previously). After that, you get the modified and you build the new Contacts file for the next time, etc.

For Deals, I don't see how to achieve the same. I make the first load with all deals, but I don't know how I can update with the new and updated deals because deals have more than one timestamp (I don't know which one to choose to set the "since" parameter).

I hope you have now more clear what I want. Do you have deeper documentation about HubSpot than the one in the web page?

Best regards.

Juan Luis

0 Upvotes
IsaacTakushi
Solution
HubSpot Employee
HubSpot Employee

Incremental upload

SOLVE

Hi, @juanluisrivero.

 

Thank you for clarifying; I believe I do have a better idea of your goal.

 

It sounds like you are retrieving contacts and deals from HubSpot, adding them to an external database, and then trying to keep the external records updated with periodic uploads.

 

If you are seeking a date property to indicate when the new and modified deals begin, I recommend using Last modified date (hs_lastmodifieddate), which records the last time any deal property was updated. This includes deal creations.

 

That said, I see a few other alternatives to your incremental upload approach. (Although I talk about deals below, the same concepts apply to contacts.):

  • Once you have uploaded the deals, you could periodically query the Get recently modified deals endpoint and page backward until you find a timestamp value you have already recorded. This approach conserves more calls than using the Get all deals endpoint alone.
  • If you create an app and connect it to your HubSpot account with OAuth, your app can notify you when contacts and deals are created or change via the Webhooks API. The Webooks API allows you to create "subscriptions" to events in accounts where your app is installed. When the chosen events occur, the application will send your service a notification with the details of the contact or deal which was created or changed. This way, you don't have to poll HubSpot's Get all endpoints for updated records.

Isaac Takushi

Associate Certification Manager
0 Upvotes
juanluisrivero
Member

Incremental upload

SOLVE

Thank you very much Isaac,

I will try with this atribute.

Best regards.

Juan Luis