Dec 6, 2021 7:05 AM
Has anyone ever witnessed the rate limit for their webhook event handlers misbehaving before?
I dealt with what equates to a DOS incident on Friday where our webhook rate limit was set to 10 per second but from our monitoring tooling it looks like for a sustained period of maybe 45 minutes to 1 hour the rate limit was ignored completely and the webhook was hit up to 300 times per second! This issue was then further exacerbated by the code in our application that allowed that event to post a message to Slack synchronously, causing some serious latency delays with handling 100k batch operations. We've since addressed that. 100k slack messages were not fun to deal with 😅
This was all triggered after a batch operation was completed on our customer data.
Dec 8, 2021 11:50 AM