APIs & Integrations

scottdoc
Mitglied

Update webhook events being sent before create events

Hello,

 

I am pretty new to the Hubspot application and community. I have inherited an integration with our system and am trying to troubleshoot some issues.

 

Our app is subscribed to both "contact.creation" and "contact.propertyChange" - however, the issue is that when a contact is newly created in hubspot, the property change events fire a few milliseconds before the contact.creation. This is causing problems in our system which dutifully handles the events in the order they were created (occurredAt) - since we cannot update the property of a contact before it is created.

 

Is this expected? 

 

Here is an cut-down example. You can see the occurredAt parameters coming from hubspot show the propertyChange events being created first.

 

{
  "eventId" : 2006566005,
  "occurredAt" : 1602139632301,
  "subscriptionType" : "contact.propertyChange",
  "attemptNumber" : 0,
  "objectId" : 2754251,
  "propertyName" : "firstname",
  "propertyValue" : "BOB",
  "changeSource" : "FORM"
},
{
  "eventId" : 3575878591,
  "occurredAt" : 1602139632301,
  "subscriptionType" : "contact.propertyChange",
  "attemptNumber" : 0,
  "objectId" : 2754251,
  "propertyName" : "lastname",
  "propertyValue" : "SMITH",
  "changeSource" : "FORM"
},
{
  "eventId" : 2937584851,
  "occurredAt" : 1602139632985,
  "subscriptionType" : "contact.creation",
  "attemptNumber" : 0,
  "objectId" : 2754251,
  "changeFlag" : "NEW",
  "changeSource" : "FORM"
}

 

6 Antworten
GabrielFuhr
Teilnehmer/-in

Update webhook events being sent before create events

Is there any update on this error? ('occurredAt' being out of order)

0 Upvotes
GabrielFuhr
Teilnehmer/-in

Update webhook events being sent before create events

Hello crtag, I would like to ask you if you were able to open a ticket related to the problem we're facing.

If you were able to open the ticket, could you please post the link here, so we can also keep track of the solution progress?

0 Upvotes
crtag
Teilnehmer/-in

Update webhook events being sent before create events

I can confirm this is happening in our integration too and I'm dealing with the implications. I'd say this is poor system design on Hubspot side and will see if it's possible to raise a ticket about this. Meanwhile, I will be delaying all incoming "update" events to ensure the entity is created properly first. This is bad idea though but will do in a short term.

btsp
Mitwirkender/Mitwirkende

Update webhook events being sent before create events

I cannot fully comment on this without HS source code access 😉 but I have faced similar issues working in various systems in the past... for example, performing TCP reassembly, and detecting when a connection should be expired / aged out to avoid overflowing the TCP reassembly buffer, or tracking the rate limit for requests expressed by an external API, and making sure my own changes do not exceed the rate limit.

 

There are two potential solutions to consider: adjustments to the HS backend which is setting the timestamps and sending change events for contact creation and property change. Arguably, the property change events should occur strictly no earlier than the contact creation event, because these event timings should be monotonically increasing. But this could be difficult to assure depending on how the system is implemented internally. Time could vary between servers, or the parts of the code generating the different events could be very widely separated from each other, and not able to synchronize the timing of the events via some kind of IPC.

 

This brings us to the solution which I have implemented in the previous use cases I described above: using a scheduling algorithm implemented with a heap sorted based on event time... when inserting into the heap... the property change events should be "penalized" a certain amount of time into the future, to prevent them from appearing before the underlying creation events are processed. Then, when reading events from the heap processing queue, in time stamp order while processing them, you can be sure it is extremely unlikely to impossible to process them in the wrong order and see an outage in the real world. The algorithm that TCP uses for sending packet retries, a token bucket algorithm, or a leaky bucket algorithm, or a task scheduling algorithm like the one in the Vixie cron source code, are all good examples of how to handle this.

scottdoc
Mitglied

Update webhook events being sent before create events

Hi @btsp 

 

Thank you for your reply. I'd written response last Friday but it looks to have disappeared. I seem to be having a hard time using this forum, I think their "spam filter" is blocking me.


The way our solution works is that each event is saved and then processed in batches of 100. I was previously ordering on the returned occuredAt value - but since that seems to be unreliable, I have now altered the code to process all create statements first, then updates, then deletes. This should hopefully prevent my system from rejecting update events that are sent before the event is created.

 

It would be good to get a take on this from someone who does have HS source access. I was confused as to whether this was a bug or intended functionality and I was missing something. Especially since their help documentation stresses that we should rely on the occuredAt value and not the order events are received (which is understandable in asynchronous, event-driven applications). However, I found it odd that I couldn't find anyone mentioning this behaviour on any forums or help documentation.

 

dennisedson
HubSpot-Produktteam
HubSpot-Produktteam

Update webhook events being sent before create events

Hi @scottdoc !

Welcome to the HubSpot Community!

@btsp , do you have thoughts on this issue?

 

Thanks!