Correct way to handle dynamic pagination in API Calls to retrieve all data

Wappler General Need Help
2

I’ve done this a couple different ways depending on the use case:

  1. I had one where we were retrieving millions of records. For this scenario you not only have to build for the dynamic nature, but also that the process may fail mid-stream and need to be restarted. I went with a combination of using bull queues to process sub-tasks of a job that is logged to the db. The “next” link was stored in the db and updated as the sub task completed in the queue.

  2. If there really is only a dozen or so possible iterations, and you want to keep it simpler, then a while loop will handle things. Here’s a very basic example of a while in server connect: How to loop in serverconnect. A while loop gives you the ability to loop a dynamic number of times, so you just have to decide when the loop should end, based on the data returned.

1 Like