How do you handle api rate limits?

Wappler General
#1

Soem api endpoints have rate limits, for example I have one with 50 requests per 5 mins.

I need to send 800 in a batch, using a server action.

After 50 requests it throws an error and my server action stops.

I need tow ait 5 min and continue.

So I need to automate:

  • Tracking from which record it should continue to send it to that endpoint
  • Plan, and execute continuation of the server action after 5 mins
  • Do this until complete

This must be very common web dev issue. How do you guys solve this?

#2

Here some pseudo code how you could handle it:

repeat (records) {
  var ok = false
  while (!ok) {
    api.send(record)
    if (api.status == 200) {
      ok = true
    } else {
      var delay = 60000
      if (api.header['retry-after']) {
        delay = api.header['retry-after'] * 1000
      }
      wait(delay)
    }
  }
}

You loop through the records. Set a variable ok to false, this will be set to true when the api call was successful. Create a while loop which will loop until the api call is ok. In that loop send the data to the api, if not ok then wait and retry. When the api sends a retry-after header use that for setting the delay otherwise use some default value like 1 minute. The code could come in an endless loop when the api send is never successful, so you probably also want to limit the number of retries.

We could probably also build the retry in the server action, does the external api return a status 429 and the retry-after header when you reach the limit?

2 Likes
#3

You could queue the requests 🦬 Bull Queues for Node

#4

Yep, this is one of the reasons I use queues as well. Just create a queue with a limiter and it will run the jobs up to the point it is allowed per minute, or whatever duration.

#5

Thanks @patrick , do you recommend to recreate this pseudocode into a wappler server action or use the Bull Queues extension?

@tbvgl @mebeingken
It automatically does the work for you to keep track of where it should continue from? And what happens in case of a server restart ors something that breaks the queue execution. Can it be resumed?

#6

Yeah, it automatically does the work for you. If the server restarts, then you need to create new queues instances for waiting and delayed jobs, and then it will just continue with the same limits.

Just make sure your redis db is persistent.

I just wrote a script that gets all the queues with waiting or delayed jobs:

async function connectToRedis() {
  const redis = new ioredis({
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_PORT,
    password: process.env.REDIS_PASSWORD,
    user: process.env.REDIS_USER,
  });

  await redis.connect();
  console.log("Connected to Redis cluster");
  
  let cursor = "0";
  const pattern = "bull:*";
  const queueNames = [];
  while (cursor !== "0") {
    const res = await redis.scan(cursor, "MATCH", pattern, "COUNT", 100);
    cursor = res[0];
    const keys = res[1];
    keys.forEach((key) => {
      const [, type, name] = key.split(":");
      if (type === "delayed" || type === "waiting") {
        queueNames.push(name);
      }
    });
  }
  
  return JSON.stringify(queueNames);
}

You can use this to trigger the script after a server restart:

#7

Awesome Ineed to put some time in this.
Last question:
There can be multiple server actions interacting with the same external API, and thus unexpectedly reaching the rate limit. How can I automatically ‘retry’?

So for example:
There’s a 50 requests/5 min limit on the endpoint.
User A: triggers server action that sends 100 requests to the endpoint. This is in a bull queue where I set a limiter to do a max of 50 per 5 mins.
User B: triggers a server action that sends 300 requests to the endpoint in the middle of user A doing this. So right now we’re at request 37/100 from user A, and the endpoint closes for 5 min
But we are adding 300 requests from user B.
So the queue of user A will stop after 50 requests, but already gets an error of reaching the max at 37 requests out of the 50 that it should do right now.

Its needs to try resume after a few min. But also perhaps in order, so first user A’s request is handled, then user B.

While typing this out, I assume I need to somehow add this all to the same queue? Or is there another solution?

PS I don’t know if the api sends a ‘retry-after’ header. Not sure how to see the headers returned from a wappler ‘api actoin’. All I can see is that the body is “retry later” @patrick

#8

The easiest solution would be to just add them all to the same queue.

You could create new queues inside a queue but that’s more complex and I’d start with the simple approach of adding them all to the same queue while you learn bull queues.

1 Like
#9

Using a queue would probably the best way, it keeps track of the jobs it already did and can continue even after a server restart. The pseudo code I gave will handle the retries and prevents the server action from stopping as soon the limit is reached, not as elegant as a queue as it will not handle restarts of the server or continue any previous failed attempts.

1 Like
#10

Thank you all, much appreciated!