APIs & Integrations

JessicaSalley
Participant

How to do an incremental load of API data pull

SOLVE

We are unable to do incremental load as we do not have (i) Ability to read the data based on date filters [Date filters are not accepted in API Call] (ii) Ability to update the existing data since we do not have unique row identifier.
Would it be possible to reach out to the support team to add some unique row identifier so that we can do insert/update. Also, if possible, some kind of date fields which we can use to extract data incrementally.
Till then we have to do full load every day (Truncate and load).

We are working with the workflows, social media, forms APIs and this is a big barrier

1 Accepted solution
IsaacTakushi
Solution
HubSpot Employee
HubSpot Employee

How to do an incremental load of API data pull

SOLVE

Hi, @JessicaSalley.

 

Thank you for clarifying. I didn't understand that you were describing issues retrieving data from HubSpot to then incrementally upload into your database. Most of the "upload" questions I field refer to HubSpot.

 

At this time, there are no plans to add date filters to the GET endpoints for workflows, social broadcast messages, forms, and form submissions. That said, I will share your use case internally so that the team knows there is need out there.

 

I also do not expect support for unique row identifiers across APIs in the foreseeable future. That said, could you generate your own using, for example, the millisecond timestamps when requests are received?

Isaac Takushi

Associate Certification Manager

View solution in original post

0 Upvotes
6 Replies 6
IsaacTakushi
HubSpot Employee
HubSpot Employee

How to do an incremental load of API data pull

SOLVE

Hi, @JessicaSalley.

 

Could you describe your ideal scenario with an example?

 

It sounds like you wish to update contacts in your account each day via either manual import or the Contacts API but are encountering issues. Am I following so far?

If so, I have two specific questions:

  1. Why do you wish to "extract data incrementally" by date? Are you seeking to check HubSpot contacts for relevant updates and then only update those contacts via import or the Contacts API? If so, could you use the Get recently updated and created contacts endpoint?
  2. Could you clarify why you do not believe you have a "unique row identifier"? If you are referring to importing objects, HubSpot supports the following identifiers:
    1. Email address or record ID for contacts.
    2. Company domain name property value or record ID for companies.
    3. Record ID for deals, tickets, and products.
    4. If you're not sure what the record ID values are, you can export the objects from HubSpot, modify the data in the rows, and re-import to update records.

Finally, if you are working out of account 1430252, you have full access to HubSpot support. Should you want it, my colleagues would be happy to walk through the import process in greater detail.

Isaac Takushi

Associate Certification Manager
0 Upvotes
JessicaSalley
Participant

How to do an incremental load of API data pull

SOLVE

Our team is working with the workflows, social media, and forms APIs for this project not the contacts API. We are working witht that instance and would appreciate a resource to be able to help if it's included in our contract. 

0 Upvotes
IsaacTakushi
HubSpot Employee
HubSpot Employee

How to do an incremental load of API data pull

SOLVE

Hi, @JessicaSalley.

 

You can contact support at any time by following the instructions on this page. Options include:

  • Live support over phone.
  • Live support over chat.
  • Support via email.

Could you clarify exactly what data you are looking to incrementally upload or update?

 

If you are working with the Workflows, Social Media, and Forms APIs, then perhaps you are looking to do all or some of the following:

It's still unclear to me how incremental uploads and row identifiers relate to these actions. To me, the term "incremental upload" invokes contacts, deals, and tickets, and I've only ever heard of "row identifiers" in the context of object imports.

Isaac Takushi

Associate Certification Manager
0 Upvotes
JessicaSalley
Participant

How to do an incremental load of API data pull

SOLVE

We are trying to pull down data on publishing and workflow goals completion into our SQL Server data warehouse for reporting purposes and blending that data with our CRM data and ecommerce and invoicing platforms. So we are not attempting to use the APIs to create content or create workflows.

0 Upvotes
IsaacTakushi
Solution
HubSpot Employee
HubSpot Employee

How to do an incremental load of API data pull

SOLVE

Hi, @JessicaSalley.

 

Thank you for clarifying. I didn't understand that you were describing issues retrieving data from HubSpot to then incrementally upload into your database. Most of the "upload" questions I field refer to HubSpot.

 

At this time, there are no plans to add date filters to the GET endpoints for workflows, social broadcast messages, forms, and form submissions. That said, I will share your use case internally so that the team knows there is need out there.

 

I also do not expect support for unique row identifiers across APIs in the foreseeable future. That said, could you generate your own using, for example, the millisecond timestamps when requests are received?

Isaac Takushi

Associate Certification Manager
0 Upvotes
KMalaiyappan
Participant

How to do an incremental load of API data pull

SOLVE

Hi Isaac,

 

I looking to get the incremental data for deal and company and the association of it. Do Hubspot have that feature in place now to make use of. If not please suggest the way to pull only the incremental data

0 Upvotes