🦬 Bull Queues for Node Version 2

[tbvgl]

@nickneustroev

1. I try to avoid that situation where one job relies on many other jobs to be finished first. Still, whenever that is unavoidable, I use my unpublished Redis extension and a counter inside Redis to track the progress and trigger the next server action when the count matches the total jobs. This is faster than tracking it via a relational database.

2. Keeping completed jobs is mostly for analytics. I would always enable remove on complete in production. If you enable it, you can define how many completed jobs you want to keep, and I use that for analytics. That way, your Redis db won’t grow endlessly, but you can still have decent analytics.

[nickneustroev]

.1.

I try to avoid that situation where one job relies on many other jobs to be finished first.

Got it, sounds reasonable. Maybe I’m gonna organize things a little bit differently later.

Still, whenever that is unavoidable, I use…

Understood. But is it possible to reach a similar result by constantly checking the status of the queue with the “Queue status” step?

.2. Got it, thank you!

.3. Also, I have a question about websockets.
Am I understanding correctly, that emitting sockets only possible in the queue that have “Same Thread” ticked?

[tbvgl]

1. Yeah, you can use the queue status step to keep track of this.

2. Yes, socket emits need to be triggered inside a same thread queue. I have a library action which adds emit jobs to a same thread queue which I’m only using for socket events.

The library action:

In the job action, I just use conditions to determine which event to emit:

So emitting an event looks like this in the end:

[htatd]

Hi @tbvgl, thanks for this update/extension! I’m using it in several functions. It works great!

I was thinking, what if I made most API’s with a bull queue job, and keep the main thread very light like that, would it make the app much more scalable and faster?

Are there disadvantages in using the bull queues for a lot of things?

[tbvgl]

@htatd

Yes, I do all external API calls and any longer running server connect actions with queues.
This allows you to control concurrency and adjust these things when you scale your server.

It also allows you to create dedicated worker apps which handle specific queues.

One example:
I built an app requiring screenshots from web pages and visual regression testing.
I tried to keep the main app as small and performant as possible, so I added the screenshot task to a queue.

This queue gets then handled by another app that only takes this queue and runs on a 128 GB memory / 64 core server.

There are no disadvantages out of added complexity, but it is well worth it.

[mgaussie]

hey @tbvgl this really is excellent - and hopefully inspire more high quality extensions from the community.

How do you handle all of your BQ’s visually? I tried to get Bull Board / other up and running without luck, so i’m building my own dash in Wappler to view, and manage.

I am finding that there seems to be a Wappler cache issue, as if I remove a job, or queue, it’s confirmed as successful, and doesn’t appear in Redis DB (I’m using a third party tool to view) but it still appears in my SC API output).

EDIT: there may also be a bug in ‘get jobs’ server action When you choose ‘Active’ it believes it’s empty, and so I get a 500 error in the console.

[mgaussie]

Hi @tbvgl getting some more errors and functions not operating like this, for other actions.

JobStatus is another I keep getting the error: “Error: parameter job id is required.” even when it’s included both in the input, as well as in the JSON file of the server action. Replacing True with False also doesn’t solve this.

PauseQueue appears to also not be working, it’s just returning a false even with the correct setup.

Cron Pattern failing: when using a cron pattern it fails with the following error: “Error: The repeat pattern must be a valid cron pattern.” even with a valid Cron pattern.

I’ve used the following without luck:
0 8 * * *
*/5 * * * *

Another: looks like parsing dynamic data into some of the bull actions causes issues. E.g. I’m parsing an interval period (as the above cron isn’t working) in milliseconds, this appears fine in the api, but when I assign this dynamic value to the interval field, the repeat does not setup correctly. If I manually type the same data, it works.

[mgaussie]

Hi @tbvgl are you planning on supporting this extension? This type of functionality is business-critical, but if it won’t be supported i’ll look for an alternative. Thanks!

[tbvgl]

Hi @mgaussie,

Thank you for your kind words and for taking the time to detail the issues you’re experiencing.

1. Visual Management of Bull Queues : I have used Bull Board and other solutions for visual management in the past. Since you’re building your own dash with Wappler, you might run into unique challenges, but it’s definitely doable.
2. Wappler Cache Issue : This is certainly an interesting issue, and it’s something I’d like to look into further. Sometimes cache inconsistencies can arise from various configurations.
3. Get Jobs Server Action & Other Errors : The error logs you’ve provided are valuable. I’ll add these to the list of items to be reviewed. Please note that bug verification and feature enhancements might take some time since this is an open-source project.
4. Support for the Extension : I am committed to supporting this open-source extension, but given its open-source nature, the pace at which bugs can be verified and resolved might vary. If you have a business-critical need for immediate or specific functionality, I offer paid support options for expedited service. Please check your direct messages for more details.

Best, Tobi

[mgaussie]

Thank you and understood.

[tbvgl]

Unfortunately, I need to discontinue this version of the Wappler Bull Queue Extension. But don't worry, it'll still be available on GitHub and NPM. However, you'll need to maintain it yourselves from now on, including submitting pull requests for any security fixes.

Current issues:

Bull Queues V3 is Almost Here

I'm really excited to let you know that Bull Queues V3 is almost done! This new version uses BullMQ instead of just Bull, and here’s why it's worth the upgrade:

• Faster Performance: BullMQ is built to be quicker and handle more jobs efficiently.
• More Reliable: It's designed to handle jobs better, so there's less chance of losing or duplicating jobs.
• Cool New Features: Supports job priorities, lifecycles, and more control over job processing.
• Flow Processor: The new version includes a flow processor for handling complex job workflows, like setting up relations where Job A handles tasks only after Job B is complete.
• Better Integration: Works seamlessly with other tools and frameworks, making complex workflows easier to build.

Why Moving On from Bull Queues V2

Continuing with Bull doesn’t make much sense now because:

1. Outdated Tech: BullMQ is a huge step up from the original Bull, so it makes more sense to use it.
2. Community Focus: The community and developers are all about BullMQ now, so you’ll get better support and updates.
3. Future-Proof: BullMQ ensures your apps are using the latest and greatest tech, keeping you ahead of the curve.

Accessing Bull Queues V3

BullMQ Queues V3 will be private and available to those who funded the project. So far, only one person has funded it. If you want to contribute financially and get access, please hit me up on GitHub Sponsorship. If you need alternative payment methods, you can DM me or email me at funding@tobi-vogel.com. I’m really sorry, but I just don’t have the bandwidth to offer it for free right now. Thanks for understanding!

I appreciate all your support and hope we can keep building great stuff together.

Cheers,
Tobi

[umadmin]

I understand the move but it’s sad. @George have you considered licensing this critical feature from Tobi? @tbvgl what sponsor amount are you requesting?