App/web server crashing - any clues

Hi,

The web serer went down earlier - under no material load. Here’s what the web server log had to say:

Any ideas ?

Thanks

pp listening at http://localhost:3000

<— Last few GCs —>

[19:0x4a62d40] 195300 ms: Mark-sweep 896.1 (1018.9) -> 896.0 (1019.6) MB, 1819.7 / 0.0 ms (average mu = 0.146, current mu = 0.001) allocation failure scavenge might not succeed
[19:0x4a62d40] 196981 ms: Mark-sweep 897.0 (1019.6) -> 897.0 (1019.9) MB, 1675.2 / 0.0 ms (average mu = 0.081, current mu = 0.003) allocation failure scavenge might not succeed

<— JS stacktrace —>
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
1: 0xa1ae50 node::Abort() [/usr/local/bin/node]
2: 0xa1b25c node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
3: 0xb9b20e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0xb9b589 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xd58815 [/usr/local/bin/node]
6: 0xd87b6a v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/usr/local/bin/node]
7: 0xd8f020 void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/usr/local/bin/node]
8: 0xd9e428 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/usr/local/bin/node]
9: 0xd80e6e v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/usr/local/bin/node]
10: 0xd81134 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/usr/local/bin/node]
11: 0xd76415 v8::internal::ItemParallelJob::Task::RunInternal() [/usr/local/bin/node]
12: 0xd76824 v8::internal::ItemParallelJob::Run() [/usr/local/bin/node]
13: 0xd904d7 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/usr/local/bin/node]
14: 0xda0a04 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/usr/local/bin/node]
15: 0xda161a [/usr/local/bin/node]
16: 0xda6897 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node]
17: 0xd64dc9 v8::internal::Heap::MarkCompact() [/usr/local/bin/node]
18: 0xd65ae3 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
19: 0xd66615 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
20: 0xd690cc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
21: 0xd36f6c v8::internal::factory::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
22: 0x1106655 v8::internal::IncrementalStringBuilder::Extend() [/usr/local/bin/node]
23: 0xe4d308 v8::internal::JsonStringifier::SerializeString(v8::internal::Handlev8::internal::String) [/usr/local/bin/node]
24: 0xe4e9d4 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
25: 0xe505c8 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
26: 0xe52f76 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
27: 0xe50987 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
28: 0xe505c8 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
29: 0xe52f76 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
30: 0xe53924 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
31: 0xc4c99b v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
32: 0x140e919 [/usr/local/bin/node]
[nodemon] app crashed - waiting for file changes before starting…

A few more errors all leading to the entire app crashing.

How buggy is the latest Wappler build proving to be ???

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (_http_outgoing.js:530:11)
at ServerResponse.header (/opt/node_app/node_modules/express/lib/response.js:794:10)
at ServerResponse.send (/opt/node_app/node_modules/express/lib/response.js:174:12)
at ServerResponse.json (/opt/node_app/node_modules/express/lib/response.js:278:15)
at /opt/node_app/lib/server.js:70:29
at Layer.handle_error (/opt/node_app/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/opt/node_app/node_modules/express/lib/router/index.js:326:13)
at /opt/node_app/node_modules/express/lib/router/index.js:286:9
at Function.process_params (/opt/node_app/node_modules/express/lib/router/index.js:346:12)
at next (/opt/node_app/node_modules/express/lib/router/index.js:280:10)
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (_http_outgoing.js:530:11)
at ServerResponse.header (/opt/node_app/node_modules/express/lib/response.js:794:10)
at ServerResponse.send (/opt/node_app/node_modules/express/lib/response.js:174:12)
at ServerResponse.json (/opt/node_app/node_modules/express/lib/response.js:278:15)
at /opt/node_app/lib/server.js:70:29
at Layer.handle_error (/opt/node_app/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/opt/node_app/node_modules/express/lib/router/index.js:326:13)
at /opt/node_app/node_modules/express/lib/router/index.js:286:9
at Function.process_params (/opt/node_app/node_modules/express/lib/router/index.js:346:12)
at next (/opt/node_app/node_modules/express/lib/router/index.js:280:10)
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (_http_outgoing.js:530:11)
at ServerResponse.header (/opt/node_app/node_modules/express/lib/response.js:794:10)
at ServerResponse.send (/opt/node_app/node_modules/express/lib/response.js:174:12)
at ServerResponse.json (/opt/node_app/node_modules/express/lib/response.js:278:15)
at /opt/node_app/lib/server.js:70:29
at Layer.handle_error (/opt/node_app/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/opt/node_app/node_modules/express/lib/router/index.js:326:13)
at /opt/node_app/node_modules/express/lib/router/index.js:286:9
at Function.process_params (/opt/node_app/node_modules/express/lib/router/index.js:346:12)
at next (/opt/node_app/node_modules/express/lib/router/index.js:280:10)

<— Last few GCs —>

[19:0x4cead40] 1095706 ms: Mark-sweep 895.4 (1017.3) -> 895.2 (1018.8) MB, 1769.2 / 0.0 ms (average mu = 0.290, current mu = 0.014) allocation failure scavenge might not succeed
[19:0x4cead40] 1097387 ms: Mark-sweep 897.4 (1018.8) -> 897.2 (1019.8) MB, 1667.2 / 0.0 ms (average mu = 0.170, current mu = 0.008) allocation failure scavenge might not succeed

<— JS stacktrace —>
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
1: 0xa1ae50 node::Abort() [/usr/local/bin/node]
2: 0xa1b25c node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
3: 0xb9b20e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0xb9b589 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xd58815 [/usr/local/bin/node]
6: 0xd87b6a v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/usr/local/bin/node]
7: 0xd8f020 void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/usr/local/bin/node]
8: 0xd9e428 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/usr/local/bin/node]
9: 0xd80e6e v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/usr/local/bin/node]
10: 0xd81134 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/usr/local/bin/node]
11: 0xd76415 v8::internal::ItemParallelJob::Task::RunInternal() [/usr/local/bin/node]
12: 0xd76824 v8::internal::ItemParallelJob::Run() [/usr/local/bin/node]
13: 0xd904d7 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/usr/local/bin/node]
14: 0xda0a04 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/usr/local/bin/node]
15: 0xda161a [/usr/local/bin/node]
16: 0xda6897 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node]
17: 0xd64dc9 v8::internal::Heap::MarkCompact() [/usr/local/bin/node]
18: 0xd65ae3 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
19: 0xd66615 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
20: 0xd690cc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
21: 0xd36f6c v8::internal::factory::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
22: 0x1106655 v8::internal::IncrementalStringBuilder::Extend() [/usr/local/bin/node]
23: 0xe4d308 v8::internal::JsonStringifier::SerializeString(v8::internal::Handlev8::internal::String) [/usr/local/bin/node]
24: 0xe4ea61 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
25: 0xe505c8 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
26: 0xe52f76 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
27: 0xe50987 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
28: 0xe505c8 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
29: 0xe52f76 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
30: 0xe53924 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
31: 0xc4c99b v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
32: 0x140e919 [/usr/local/bin/node]
[nodemon] app crashed - waiting for file changes before starting…

And again. Basically this app has ground to a halt. GAME OVER

App listening at http://localhost:3000

<— Last few GCs —>

[19:0x417bd40] 1164229 ms: Mark-sweep 904.4 (1019.9) -> 904.4 (1019.9) MB, 1684.1 / 0.0 ms (average mu = 0.080, current mu = 0.000) allocation failure scavenge might not succeed
[19:0x417bd40] 1165937 ms: Mark-sweep 904.4 (1019.9) -> 904.4 (1019.9) MB, 1707.8 / 0.0 ms (average mu = 0.041, current mu = 0.000) allocation failure scavenge might not succeed

<— JS stacktrace —>

==== JS stack trace =========================================

Security context: 0x355c745008d1
0: builtin exit frame: stringify(this=0x355c7451ee79 ,0x37012e8404b1 ,0x37012e8404b1 ,0x0e56061b4579 ,0x355c7451ee79 )

1: arguments adaptor frame: 1->3
2: stringify(aka stringify) [0x20bcd1991519] [/opt/node_app/node_modules/express/lib/response.js:1150] [bytecode=0x37ecc6e05...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xa1ae50 node::Abort() [/usr/local/bin/node]
2: 0xa1b25c node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
3: 0xb9b20e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0xb9b589 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xd58815 [/usr/local/bin/node]
6: 0xd58ea6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
7: 0xd65765 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
8: 0xd66615 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0xd690cc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0xd36f6c v8::internal::factory::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
11: 0x1106655 v8::internal::IncrementalStringBuilder::Extend() [/usr/local/bin/node]
12: 0xe4f8b1 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
13: 0xe52f76 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
14: 0xe50987 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
15: 0xe505c8 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
16: 0xe52f76 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_(v8::internal::Handlev8::internal::Object, bool, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
17: 0xe53924 v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/usr/local/bin/node]
18: 0xc4c99b v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
19: 0x140e919 [/usr/local/bin/node]
[nodemon] app crashed - waiting for file changes before starting…

The only thing I can see in the information you given is that each time it crashed because the app was out of memory. I can’t see in the stack traces what is causing it. Do you have some very large datasets or doing some endless loops? Does it happen when calling a specific server action?

You can try to increase the memory limit …

Yes, something has clearly started grabbing al the available memory. We could try increasing the allocation but that’s just masking the issue

The app has been running as usual. I’ve upgraded my Wappler build recently and crashes have started. We don’t have endless loops and while we do have large datasets, they’re carefully managed when querying and haven’t changed lately anyway.

We can’t pinpoint when it happens, it doesn’t take very long after restarting now. Usually a 10-minute window.

It looks like we will have to run a node profiler …

If you are using a shared server, it is possible to have such problems.

From which version of Wappler did you upgrade, then I can take a look what changed between that version and the current one. It would certainly help if we could figure out which specific action is causing the issue.

Does it also happen on the development pc? Try to monitor the memory usage while trying out different server actions.

Agreed. however we’ve not had these issues to date. Now they’re consistent and crashing within minutes of a restart under normal usage - feels like a recent change somewhere in the stack is causing a serious memory leak

It was only about 6 weeks since we last upgraded. We don’t run on local machines - Docker desktop on Windows is a dog.

We’ll try profiling and see what clues we get

Everything is clearly written in the error message above. Memory is not enough right now.
If there was such a problem , many people could have written the same error .

My idea is to first increase the memory in docker side to solve the error message and do the same for nodejs. Then you can try again.

I fully agree it’s a lack of memory - but adding more memory won’t solve the underlying issue - It will simply mask the problem until a later time when we have more users on the system. At this stage, we need to find the leak as it’s just started happening and there’s only a tiny number of users.

Well maybe you are suddenly processing large amounts of json data? Try to isolate when is the crash happening during what action steps?

I will increase memory as a first step. It looks like the memory limit needs to be specified when the node service is started using the ‘–max-old-space-size’ switch. I’m guessing that needs to be set up as part of the docker deployment. How would you recommend doing so?

Maybe this will help: