"JavaScript heap out of memory" with big task

I’m building integrations to sync API data to my DB. It works all fine with “small” amount of records (1K-10K) but when I tried to do a large set it crashed the app. I see that it crashed when it came to 13.700 records.

My server connect api is basically first looping the api to get all records, and when that finishes I’m repeating all the loaded api records to insert it to the db. It reached the inserting to the DB part and crashed there so I’m assuming that it was just too “heavy” to repeat / do stuff / insert into the db.

Does anyone have any advise on how to solve this? I’m reading on Google that I should set a higher memory limit to node to process the task, but I’m planning to run such tasks often, will it affect the main app processes/functioning? How can I make this more efficient?

This is the server connect api:

The error:

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb02930 node::Abort() [/usr/local/bin/node]
 2: 0xa18149 node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0xcdd16e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0xcdd4e7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0xe94b55  [/usr/local/bin/node]
 6: 0xea481d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0xea751e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 8: 0x121701d v8::internal::Deserializer<v8::internal::Isolate>::Allocate(v8::internal::SnapshotSpace, int, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0x121b283 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject(v8::internal::SnapshotSpace) [/usr/local/bin/node]
10: 0x1219c40 v8::internal::Deserializer<v8::internal::Isolate>::ReadObject() [/usr/local/bin/node]
11: 0x12144f8 v8::internal::ContextDeserializer::Deserialize(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [/usr/local/bin/node]
12: 0x1214676 v8::internal::ContextDeserializer::DeserializeContext(v8::internal::Isolate*, v8::internal::SnapshotData const*, bool, v8::internal::Handle<v8::internal::JSGlobalProxy>, v8::DeserializeInternalFieldsCallback) [/usr/local/bin/node]
13: 0x1239ce8 v8::internal::Snapshot::NewContextFromSnapshot(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSGlobalProxy>, unsigned long, v8::DeserializeInternalFieldsCallback) [/usr/local/bin/node]
14: 0xf489e5 v8::internal::Genesis::Genesis(v8::internal::Isolate*, v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
15: 0xf49818 v8::internal::Bootstrapper::CreateEnvironment(v8::internal::MaybeHandle<v8::internal::JSGlobalProxy>, v8::Local<v8::ObjectTemplate>, v8::ExtensionConfiguration*, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
16: 0xcea976 v8::NewContext(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, unsigned long, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
17: 0xceae44 v8::Context::New(v8::Isolate*, v8::ExtensionConfiguration*, v8::MaybeLocal<v8::ObjectTemplate>, v8::MaybeLocal<v8::Value>, v8::DeserializeInternalFieldsCallback, v8::MicrotaskQueue*) [/usr/local/bin/node]
18: 0xaf33f1 node::contextify::ContextifyContext::CreateV8Context(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [/usr/local/bin/node]
19: 0xaf41d7 node::contextify::ContextifyContext::ContextifyContext(node::Environment*, v8::Local<v8::Object>, node::contextify::ContextOptions const&) [/usr/local/bin/node]
20: 0xaf47d9 node::contextify::ContextifyContext::MakeContext(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
21: 0x1569fec  [/usr/local/bin/node]
[nodemon] app crashed - waiting for file changes before starting...

You could rework your steps to only fetch the first 10.000 records per execution. Maybe you can save the offset in the database, so the next run it starts from such saved offset :wink:

I had a similar issue, so that’s what I would do. In reality, I ditched Wappler and went for n8n or a hand-coded script :grimacing: You could also do the processing in a custom formatter/module and then just return the big array to Wappler, so you could use a Repeat step to Database insert

Hi!

My “integrations module” is for end-users in my app that are connecting to the apps they use to sync their data to my app, so it’s heavily based on DB records based on their info. Creating a custom module / hand-coded script would be wayy above my knowledge level :sweat_smile:

So in the server connect api it fetches all api data without a problem, otherwise it wouldn’t reach the “inserting” part.

Maybe instead of fetching all, then inserting, I could fetch 1K, insert 1K, and continue etc? Would that free up the memory with every repeat? Or does the memory only free up after finishing the entire server connect api?

Ok so this worked now easily with 34K records. I don’t know why I wanted to fetch all and insert all after in the first place :stuck_out_tongue_winking_eye:

Is anyone interested in a tutorial on how to make a native integrations module for end-users? My set up is quite neat right now, with the db tables set up and modularised server connect api’s I can set up now any native integrations easily and end-users can sync data for their users, perfect for B2B SaaS applications

1 Like

@karh might be interested

Glad you got it working :slight_smile:

A difficult question to answer, because Wappler might keep appending on every iteration of the Repeat so it can output the final response in the end. We can only hope NodeJS’ garbage collection runs fast enough to prevent issues, and looks like it does since you were able to run it without problems :slight_smile:

2 Likes

Definitely interested in such a tutorial :smiley: !

Thanks @Apple