🦬 Bull Queues for Node Version 2

[tbvgl]

Done @karh

Best first to remove the V1 extension.
Then just run npm install wappler-bull-queues or use the Wappler extension installer.

If you run into any issues with the Redis connection then best define the Redis variables via ENV
https://github.com/tbvgl/wappler-bull-queues#optional-env-variables.

The bull extension automatically installs the advanced logger as well in case you did not install that one, yet. Logging is configured via ENV variables as well. If you just want pretty console logging add this to your ENV:

PRETTY_LOG=enabled
LOG_LEVEL=debug

You can find more options here:

GitHub - tbvgl/wappler-advanced-logger: Advanced Logger Extension for...

The Bull extension has logging integrated, so you will see logs immediately as soon as you run some jobs or create queues.

[karh]

Thank you so much @tbvgl - super excited to try it out. Hopefully today. Will update.

[karh]

@tbvgl Is this based on BullMQ?

[karh]

@tbvgl Sadly getting this error:

[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:6379
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) 

I installed using npm.

@tbvgl

Since the new update, there was still an issue with ioredis and how Wappler handles the Redis client config. I pushed another fix for V2 of the bull queues extension. The issue led to the usage of 127.0.0.1 for the host in some instances as well.

I’m not sure if you are using V1 or V2 of the bull extension @umadmin, but if you are using V2, then you only need to update the bull extension.

I’m guessing this should be fixed, isn’t it the issue from this thread?

EDIT Fixed by adding the REDIS_HOST config var

[umadmin]

Preformatted textAfter pulling the latest update I’m seeing

[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:6379 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)

again. I ensured the sockets.js matched was was in Socket Error in Library Actions Any ideas or what to look at to debug?

[tbvgl]

@karh are you still getting errors after defining the creds via ENV?

@umadmin did you try using env variables for your connection parameters?

I think I’m going to depreciate connection parameters from globals and only allow ENV creds in the future.

[tbvgl]

server_connect.zip (15.0 KB)

@karh can you test if this one works for you without env variables?

[tbvgl]

no, OptimalBits/bull because MQ requires typescript.

[umadmin]

I’m not using environment variables. Can you provide what the parameters should be if leveraging the standard wappler config?

[tbvgl]

@umadmin You will need to use ENV variables as documented in the GH repo or you can try this fix for global configs from Wappler:

Bull Queues for Node Version 2

server_connect.zip (15.0 KB) @karh can you test if this one works for you without env variables?

[umadmin]

That’s my question though. If leveraging the default wappler redis config what should the settings be for things REDIS_PORT, REDIS_HOST, REDIS_USER, etc. Which are required?

[tbvgl]

That depends on your Wappler redis config. Can you share the connection string from server connect?

I think the current default if you manage redis via Wappler would be just:

REDIS_HOST=redis

[umadmin]

Yes this is the default

[tbvgl]

Yeah then you can use the ENV from above or try the zipped code.

[umadmin]

The zipped code fixed the error. I also added the environment variable

[karh]

Sorry gotta go, but it seems that umadmin tested it

Appreciate the quick support on your extension btw

[tbvgl]

Thanks guys, I pushed V1.8.3 with the fix https://www.npmjs.com/package/wappler-bull-queues/v/1.8.3

npm update or using the Wappler package update button will automatically install the newer version.

[tbvgl]

Btw quick reminder here: If you use queues in sandboxed threads then every new job will create a new DB connection. They get automatically destroyed when the job is done. But if you run a lot of concurrent jobs in sandboxed threads and use Postgres then make sure to put pgbouncer in front of it so that you don’t run out of connections. That way you can easily handle thousands of concurrent jobs if your server has the power.

[nickneustroev]

@tbvgl Thanks for your amazing work!

I have a couple of questions and I will be grateful if you share your thoughts about it.

1. What do you suggest as the best practice for catching the end of the processing all the job? (for triggering another server action)
   For now I just mark the last job with a special parameter. It works, but obviously is not error-prone.

2. Is it worth it to tick the “Remove on complete” on Job Start?
   Currently I have it in default state (empty).
   Does this mean that all completed Jobs will be stocking in Redis and in some time can slow it down?