Correct way to handle dynamic pagination in API Calls to retrieve all data

Wappler General Need Help
1

Hi all,

I have a scenario where an API we are using will only return 10 results per call. The amount of results will vary from time to time. If there are more results they provide a unique string that we can use to call the next set - all pretty standard.

However, how should I handle the dynamic nature of this in our Server Connect workflow. Right now, I call the API, I have a condition that checks the api response if there is data in a ‘next’ field (the unique string). I then process another call. I then have a condition that checks that response, and so on.

The issue is with this workflow, it’s not really handling the amount of results dynamically. I have 3 conditions currently, which mean a maximum of 30 results, I could have 10 for 100 results etc. but that seems quite messy.

I will then need to join all the results somehow, again, with them being dynamic that’s a little difficult.

I know we have Repeat, and While but i’m a little stuck on the best, most efficient and reliable way to handle this.

Appreciate any guidance here.

2

I’ve done this a couple different ways depending on the use case:

  1. I had one where we were retrieving millions of records. For this scenario you not only have to build for the dynamic nature, but also that the process may fail mid-stream and need to be restarted. I went with a combination of using bull queues to process sub-tasks of a job that is logged to the db. The “next” link was stored in the db and updated as the sub task completed in the queue.

  2. If there really is only a dozen or so possible iterations, and you want to keep it simpler, then a while loop will handle things. Here’s a very basic example of a while in server connect: How to loop in serverconnect. A while loop gives you the ability to loop a dynamic number of times, so you just have to decide when the loop should end, based on the data returned.

1 Like
3

Thanks Ken, with this I got a basic loop up and running, and set a global value for if the next token exists.

A question, how did you handle the API responses? Inside my loop I have the API, but as it loops, it’s just returning the new set of 10 each time. I could loop > insert 10, then loop again and continuously do it this way.

4

Exactly, just store the results in the db for each loop.

1 Like
5

Thanks Ken, this looks like a good solution. Appreciate it.

1 Like