App triggers two webhook events for one propertyChange

[igor_kurochka]

Hi, we have a weird behavior of our app integration since yesterday.

from around 10/10 12pm it started to send two webhooks for a single propertyChange.

The payload is all the same except "eventId" only which is different. But I'm 1001% sure that there is only one change happened, why does it generate two events and sends two webhooks?

Does anyone experience anything similar? what could be a reason and how to stop this?

[lscanlan]

Hey guys,

I've seen a few reports of this today. It looks like what's happening is that property changes on deal objects are triggering duplicate webhook payloads to be sent out. Like you said @igor_kurochka, the payloads are identical (including timestamps) except for the eventId. The payloads are also batched together. From what I've seen it looks like this is only happening on deal objects and not contacts or companies.

I've passed the issue along to our Engineering team. I'll report back when I hear more. If anyone sees behavior different from what I described here, please let me know.

[lscanlan]

Hey, I just wanted to update this thread for anyone watching it. The latest update from today is that we've identified what's causing the duplicate notifications. We should have a fix for it within a day or two.

[gotmike]

watching this thread b/c we are having the same issue, do u have an updated timeline for fixing this?

[lscanlan]

Hi @gotmike, So the duplicate notifications were caused by a CRM service doing a kind of back-end migration for deals. Because of the migration, that service was sending out duplicate deal messages, which the webhooks service wasn't deduplicating properly.

That migration is now complete, so the duplicate webhook notifications should have stopped by now. In the meantime, the webhooks team has also been working on improving the deduplication process. They have a fix that's being tested, and I believe the plan is to push it live today. That will prevent a situation like this from happening again. I'll update again here when that's live, but by now the dupliate webhook notifications should already have stopped.

[lscanlan]

Hi everyone,

The fix I described ^above is live. One thing to point out here is that in our Webhooks documentation here: https://developers.hubspot.com/docs/methods/webhooks/webhooks-overview, there is a note in the payloads section:

NOTE ON UNIQUENESS: We do not guarantee that you will only get a single notification for an event. Though this should be rare, it is possible that we will send you the same notification multiple times.

So the mass duplicate notifications should be fixed now. But it is possible for duplicates to happen, albeit rarely, and certainly nowhere near the scale that was happening here.

Let me know if anyone has questions.

[yadewale]

@igor_kurochka, I experienced the exact same issue just yesterday and I've reported the issue but no response so far.