Running tasks in parallel

Wappler General How To
#1

Hi all,

I have a workflow that repeats data from the database and then on each row calls an external API that can take 1 - 2 seconds to respond. This is causing some timeout issues when I want to handle more records at once.

It looks lie the Parallel module won’t help here, as it can’t handle working with an array. I’m wondering how everyone here handles running multiple actions/api calls etc. in parallel with Wappler?

This will drastically improve the speed and performance of 3 core api’s in my new project.

#2

Maybe Bull Queue? @mebeingken or @tbvgl might be able to confirm if this would be helpful for you.

#3

Yes, that’s a good use case for the Bull extension.

#4

Thanks @kfawcett unfortunatelly that custom extension in particular has too many bugs/issues with it to be confident in using it. BQ’s are were I headed initially, perhaps i’ll be forced to build my own extension on the package if no other solution presents itself.

#5

I’m using that extension in many api’s, never had a problem. What bugs did you experience?

1 Like
#6

Lots unfortunately, but to name a few;

  • Adding a working api (that works 100% perfectly using Wappler’s scheduler) to a job will in 90% of cases just sit at delay, and then eventually fail or just sit on delay.
  • Recurring jobs don’t seem to work at all.
  • {status: “500”, message: “jobState is not defined”,…} error, when the job is clearly defined in the UI. This happens across most of the modules/functions provided in the full extension.
  • Seems to be some caching issues (as discussed in release post).

etc…

#7

Can you share how you’re using it? I think you’re not creating the queue when needed, and the queue needs to be on autostart.

Can’t comment on this as I’m not using it, perhaps you can do the recurring jobs with Wappler’s scheduler

#8
  1. Delayed jobs work
  2. Recurring jobs work
  3. Job state - somewhat a bug because it should display the job with the ID is non existent instead of displaying this error

So overall no major extension bugs. I highly recommend studying the bull docs to get familiar with how everything works.

#9
  1. Incorrect, as the job should not be delayed, it is an active job that should be run immediately as there are no competing jobs or workers and is defaulting to ‘delay’ and just gets stuck there. The ‘action’ works standalone, or in a scheduler with Wappler but not with BullQ’s. I’ve attempted to add as a single job to a q, and as a recurring job.
  2. Nope, not for me during during my testing.
  3. Yeah, so it’s a bug.

I am unable to run simple test ‘jobs’ which is the core purpose of the extension, right? I guess I’ll leave the interpretation of ‘major’ bug up to others.

I’m familiar with the documentation and how the bullq system should work, thank you, and I also have my trusty ChatGPT to dumb things down for me.

I’m not too interested in getting into a discussion around it, my point was that I have been unable to use this solution due to he bugs that I found, which is unfortunate due to the amount of time and effort that no doubt went into the initial build. I’m glad others have found it to be more reliable and useful as that’s good for the community.

#10

It’s been a while, but from memory I was with every job creating the queu again, as if the queue is created already it will just ignore that and add it. I played with both autostart and non autostart.

The implementation followed the guidance from the documentation - I mainly had issues with jobs being added, and then just getting stuck at delay, with no reason for it - that plus some caching issues. I could work around everything else - I just needed to reliably create queues and add jobs with pre-existing working actions.

It’s clearly the right solution so i’ll have to return to it next week and either build fresh, or, try to make improvements to get this working. I appreciate your responses though!

#11

I don’t know about the caching issues as I don’t use that, but I’m sure you can make it work as I’m using it reliably in a lot of server actions, good luck!

1 Like
#12

Just a tip to get you going:
Your 1. and 2. issues both sound like there was no worker attached. So make sure a worker is attached when adding the jobs.