APIs & Integrations

GKeane
Membre

Misbehaving Webhook ratelimiting

Hey folks,

Has anyone ever witnessed the rate limit for their webhook event handlers misbehaving before?

I dealt with what equates to a DOS incident on Friday where our webhook rate limit was set to 10 per second but from our monitoring tooling it looks like for a sustained period of maybe 45 minutes to 1 hour the rate limit was ignored completely and the webhook was hit up to 300 times per second! This issue was then further exacerbated by the code in our application that allowed that event to post a message to Slack synchronously, causing some serious latency delays with handling 100k batch operations. We've since addressed that. 100k slack messages were not fun to deal with 😅

This was all triggered after a batch operation was completed on our customer data.

0 Votes
1 Réponse
dennisedson
Équipe de développement de HubSpot
Équipe de développement de HubSpot

Misbehaving Webhook ratelimiting

@GKeane 

oof.  I have  not seen this before.  I assume it has passed.  Please do report if it comes back. 

0 Votes