Managing Rate Limiting in Workflows with Custom Code Actions

Developer Support APIs & Integrations
6

Hi @pascalkremp and thanks @DianaGomez for the tag

Pascal we faced a similar problem in various use cases where workflows and apps are limited by 3rd party rate limits. Here are the principles of our approach if you are interested in coding a similar solution:

  1. Add a middleware/proxy layer where all calls to the 3rd party API can be centrally throttled. As middleware, the workflow will call your API endpoint, which calls the API on behalf of that workflow instance, then returns the result.
  2. Within the middleware, implement token-bucket rate limiting. Set the depth of the bucket and the rate at which tokens are added until the bucket is full, for example, 1 token per second, to suit your 60 per minute use case. Your code to refill the bucket is only required when the bucket is not full.
  3. When your middleware receives a call from your workflow, it must first request one or more tokens for immediate use from your token bucket code. If token(s) are granted, the API call is made and the result returned to the original caller. If no tokens are available, have your middleware pause up to a set limit before returning a failure message to the calling code, ideally within HubSpot’s 20-second workflow action timeout.

Levers you can adjust when tuning your token bucket to the external API include:

  • Token Bucket Depth
  • The rate at which tokens are re-added to the bucket
  • Max number of tokens in a single token request (to avoid one caller hogging all tokens)

This approach is similar to the technology employed in internet routers for throttling demand from many users over a network connection of limited bandwidth.

This approach should allow all your workflow requests to be shaped to fit the maximum allowed rate from the 3rd party API.

You can add more advanced techniques to this base method, depending on your use case:

  1. Create a token bucket per 3rd part API key. Each key increases your available rate limit, e.g. two keys to 2 x 60/minute. Internet routers use this approach for load sharing traffic across multiple network links to the same destination. Add a another link or API key to add more bandwidth.
  2. Create multiple token request queues for requesters of different priority calling the same 3rd party API. High priority workflow actions make high-priority token requests. Normal workflows make normal priority token requests. Your middleware always serves priority requests before normal requests. Again this approach is used for priority network traffic, such as delay-sensitive voice packets being routed before lower priority web/email traffic.

We use this approach at HubDo in various forms, as middleware for workflows or within an app. It’s a very deterministic approach for managing demands from multiple discrete requesters to a single limited resource.

Interested to hear what others have done to solve rate limiting in a deterministic way.

best

Pete