I'm working with a data warehouse which depends on a synchronization with the HubSpot API, and I'm trying to get data from the engagements API. It's known that the Contacts, Companies, Deals and Tickets APIs are very offset dependant - the offset can change if some record is changed/deleted. However, the engagement record is not deleted (as far as I know). In this case, could offset be used as a checkpoint? I'm only asking this since there is a large amount of data retrieved from this specific API, and hence the full load would not be viable for regular environments (non Big Data environments).
I wouldn't recommend using the offset value as a checkpoint and this is because while the enegagement record may not have been deleted thus far, there is a possibly that it would be deleted someday and this is because engagement such as notes/tasks can be deleted.
In this case, I'd suggest using the Get recent engagements to retrieve and store engagements into your external software/database instead.
I mean, it would be great if I only needed the most recent engagements. However, I need to get all the records once, so I could then keep the sync via the most recent engagements endpoint. Have you got any suggestion for that?
In this case, my best suggestion is for you to run the Get all engagements over non working hours so that users are not actively creating engagements, you'll only have to loop through this endpoint once.
Moving forward, you can use the Get recent engagements endpoint to up keep the engagements.
Yes, that would. However, I won't be able to catch the event of the deletion of the engagement, so my data warehouse may always contain noisy data which was already deleted. I'm saying this based on he docs of the engagements endpoints and on the Webhooks API, where you can't subscribe anything different than Deals, Contacts and Companies.
I understand your use case for this request and if you haven't already, I'd recommend jumping over to the HubSpot Ideas Forum and posting this idea there so that it's visible to the HubSpot product team and other HubSpot customers and developers. Hopefully this will gather more interest.
.png "HubSpot Employee"){kind=icon}
