Misbehaving Webhook ratelimiting

GKeane
Member

Hey folks,

Has anyone ever witnessed the rate limit for their webhook event handlers misbehaving before?

I dealt with what equates to a DOS incident on Friday where our webhook rate limit was set to 10 per second but from our monitoring tooling it looks like for a sustained period of maybe 45 minutes to 1 hour the rate limit was ignored completely and the webhook was hit up to 300 times per second! This issue was then further exacerbated by the code in our application that allowed that event to post a message to Slack synchronously, causing some serious latency delays with handling 100k batch operations. We've since addressed that. 100k slack messages were not fun to deal with 😅

This was all triggered after a batch operation was completed on our customer data.

0 Upvotes
1 Reply 1
dennisedson
Community Manager

@GKeane 

oof.  I have  not seen this before.  I assume it has passed.  Please do report if it comes back. 

Thanks,

Dennis




Check out our Community Developer Blog
where we feature our Community driven developer podcast and how to content
0 Upvotes