NodeJS server crashes a lot on redis during disconnect

Wappler General Bugs
1

Wappler Version: W7B16
Operating System: W11Pro
Server Model: NodeJS

[nodemon] starting `node ./index.js`
App listening at http://localhost:3000
[ioredis] Unhandled error event: Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Error: delete from "sessions" where expired < CAST($1 as timestamp with time zone) - Connection terminated unexpectedly
    at Connection.<anonymous> (D:\project\node_modules\pg\lib\client.js:131:73)
    at Object.onceWrapper (node:events:632:28)
    at Connection.emit (node:events:518:28)
    at Socket.<anonymous> (D:\project\node_modules\pg\lib\connection.js:62:12)
    at Socket.emit (node:events:530:35)
    at TCP.<anonymous> (node:net:337:12)
Connection Error: Connection ended unexpectedly
[ioredis] Unhandled error event: Error: read ECONNRESET
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Connection Error: Connection ended unexpectedly

The NodeJS app crashes with errors like above randomly. Sometimes while I am working in Wappler, sometimes when I am not. I think it could be because of network changes.. I enable/disable VPN connection few times as I work on multiple things.
Maybe there needs to be a better exception/network-changes handling?

2

Same here

4

The error is from Redis loosing its connection unexpectedly. Do you use a remote Redis server which looses the connection when you disable your VPN?

5

@patrick the Wappler ioredis config is very basic

config.js

if (config.redis) {
    const Redis = require('ioredis');
    global.redisClient = new Redis(config.redis === true ? 'redis://redis' : config.redis);
}

Maybe add some debugging logs and a reconnect strategy like

if (config.redis) {
    const Redis = require('ioredis');
    
    const redisOptions = {
        retryStrategy: (times) => {
            const delay = Math.min(times * 100, 2000);
            return delay;
        },
        reconnectOnError: (err) => {
            if (err.message.includes('READONLY')) {
                return true;
            }
            if (err.message.includes('ECONNRESET')) {
                return true;
            }
            return false;
        }
    };

    global.redisClient = new Redis(
        config.redis === true ? 'redis://redis' : config.redis,
        redisOptions
    );

    global.redisClient.on('connect', () => {
        console.log('Redis connected successfully.');
    });

    global.redisClient.on('ready', () => {
        console.log('Redis is ready to use.');
    });

    global.redisClient.on('error', (err) => {
        console.error('Redis error:', err);
    });

    global.redisClient.on('reconnecting', (delay) => {
        console.log(`Reconnecting to Redis in ${delay}ms...`);
    });

    global.redisClient.on('end', () => {
        console.log('Redis connection has been closed.');
    });
}
6

A reconnect strategy is indeed a good thing to have. Another thing is that nodemon in doesn't restart the server on a crash, it is nice that it reloads on file changes but crashes will just freeze it until you update some file.

7

Good point! Nodemon can be configured to restart the server after a crash.

nodemon --exitcrash server.js

Then in the nodemon.json (config)

{
  "watch": ["server.js", "config/*"],
  "ext": "js,json",
  "exec": "node server.js",
  "exitcrash": true
}

And in package.json

{
  "scripts": {
    "start": "nodemon --exitcrash server.js"
  }
}

Basic implementation and just an idea.

8

Redis is remote, but its not dependent on VPN. It can work with VPN on/off.
After crash, the server does not restart even after saving a file. I have to manually restart the server.

9

Hi @patrick
Any update on this?
I still frequently see this happening in Beta 17, and even when there is no network change. There is no error in such cases. When there is a network change, the error usually is from redis package.

10

Do you know from which version of Wappler this issue started?

11

I only started using Beta regularly from v15. So not sure if this has been so in previous betas.

12

You only have this on the Beta and it didn't happen on the latest stable version?

13

For stable it does crash in case of network change. I think it has been like so for years.
But otherwise, there is no issue.

Because beta was crashing so much, I noticed the redis error.

14

Bump.

15

I've added the retry strategy to the Redis config as @tbvgl suggested.

redis.zip (633 Bytes) unzip to lib/setup

2 Likes
16

This seems to be working well so far in case of network change, and redis errors.
Will have to see if the random crashes are happening as well or not.

17

image
image1482×328 26.9 KB

Was on a client call and hadn't changed network, nor made any changes in Wappler. Wappler was just open in the background.
Coming back to Wappler post call, I see this. There are two instances of redis connection errors.. but no log after that, and the app is in crashed state.

The new redis fix would have worked when redis threw an exception, but the underlying problem still remains.

18

@sid can you check if maybe your redis connected clients keep increasing and are you using bull queues in that project?

You can check connected clients in redis with ‘client list’

19

I am using bull queues, but while on localhost, there is barely any queue related activity.
I will try to check the connected clients when the app crashes, and get back to you.