Bulk question

[franse]

I have a vps which queue all the jobs..

Backend is running in api.domain.com and let's say the service is running in jobs.domain.com
Everything is doing great here

But I need to know how to handle this situation I was asked (Using this an example):

If I need to send 10.000 jobs, I can't call the endpoint 10.000 times, so asked AI and it suggested do it by sending all of them at once to another endpoint:

{
  "job": "sendReminder",
  "items": [
    { "user_id": 1, "phone": "..." },
    { "user_id": 2, "phone": "..." }
    // ...
    // 10.000 items
  ]
}

That's not a complicated part.

But suggested another professional scenario of sending batches, 1000 x 10 times.
How can I do this using Wappler? I was thinking about a paged query since users come from db

Is there another way I can do this? I hope I've made myself clear and thanks

[Apple]

@franse

I was thinking about a paged query since users come from db

Yes, you know the number of rows, divide by 1000, use a Repeat step to repeat n number of times you calculated, use Database Paged Query step inside, instead of getting pagination from some $_GET param you use a variable you build, you have access to current iteration count, and you can multiply by 1000 if needed.

This is probably called chunking, or cursor iteration, or something like that…

[Hyperbytes]

Alternative you could insert them into a temporary table and use a scheduler to do them in small batches at time intervals.

[franse]

@Hyperbytes

Alternative you could insert them into a temporary table and use a scheduler to do them in small batches at time intervals.

Yes, thought about that alternative, but I have an external endpoint where I can send them at once, so staying on that, thanks Brian

@Apple

Yes, you know the number of rows, divide by 1000, use a Repeat step to repeat n number of times you calculated, use Database Paged Query step inside, instead of getting pagination from some $_GET param you use a variable you build, you have access to current iteration count, and you can multiply by 1000 if needed.

Thanks @Apple for confirm that, made a test and it's working

Thanks both!