JavaScript heap out of memory advice

Wappler General Need Help
1

As this is going to be a long question, I will start at the end.

I have a script, with a few custom modules i have written, that when run, after about 2 minutes trows the following error.

<--- Last few GCs --->

[1:0x789e000]   160886 ms: Scavenge (interleaved) 967.1 (992.5) -> 963.6 (994.7) MB, pooled: 0 MB, 30.27 / 0.00 ms  (average mu = 0.354, current mu = 0.353) allocation failure; 
[1:0x789e000]   161486 ms: Mark-Compact (reduce) 972.1 (997.6) -> 962.3 (980.8) MB, pooled: 0 MB, 12.50 / 0.00 ms  (+ 515.7 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 588 ms) (average mu = 0.352, current mu 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xe2f0f2 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0x1240cf0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0x1240fc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x1470715  [node]
 5: 0x1470743  [node]
 6: 0x14897fa  [node]
 7: 0x148c9c8  [node]
 8: 0x1e25421  [node]

I realise that the first thought is to increase the memory with something like NODE_OPTIONS=--max_old_space_size=2048 however, this error happens after inserting 59 000 records into my database, and the amount of records I need inserted is a whopping 2.5 million, so I do not think increasing the memory is going to work, unless i gave it 1tb of memory.

What and Why?
I am connecting to an accommodation provider, who has an API, and their system asks that all hotel searches are performed by the end user, and their API is only responsible for giving single hotel live rates and availability once the user has chosen from the 25 search results provided on my site.

So they give me an API endpoint that passes back a URL to an AWS server where I can download a compressed file that holds all 2.5 million hotels they manage rates and availability for. I am meant to download the 2gb .zst file, decompress it back to a 28gb .jsonl file, and use the data contained to setup my own hotel search dashboard with filtering etc.
Once the user has chosen a hotel to view, that is when i start pulling single hotel live rates and room availability from their live API endpoints.

My Solution
My server is a Digital Ocean Docker 4gb memory, 100gb disk, 2 vCPUs, Node JS

I have a server action that looks like this

  1. Call their API and get the download link
  2. Download the file and add the .zst file to my Docker Volume using custom module from Hyperbytes
  3. Decompress the .zst into a .jsonl file with my own custom module which takes about 4.5 minutes and adds it to the same path as the .zst
  4. Add the .jsonl file location to a database table as a single row
  5. Another of my custom modules then reads the entire .jsonl file line by line and stores all 2.5 million byteLengths in a single array
  6. Database Bulk Insert step takes the array and adds all to a second database table of 2.5 million rows, in 34 seconds.
  7. Custom Database Query to output the first row with a byteStart of 0 and every 300th row byteStart. Which returns 8620 ByteStart integers
  8. Repeat step
  9. Custom Module by myself and Hyperbytes, uses fs.createReadStream(load_path, {start: startBytes, end: endByytes}); to save reading the entire 28gb .jsonl file, it only reads about 30mb and fetches 300 lines, then adds each line to an empty array.
  10. Database Bulk Insert to a third table which stores a line per row in its original JSON format.

And that is basically the entire flow, when i run this, it is very fast, and in roughly 2 minutes I have 59 000 records inserted into my 3rd table, but then it runs out of memory and just dies with the error message above, this means it has only run the repeat 196 times out of the total 8620 times it needed to complete the task.

So question is how do i overcome this, is there a way to release the memory once it has been written to the database, or can i use some kind of bull queries from @mebeingken to try split this up somehow, just not sure what to do from here, considering its working, sort of, lol.

2

You're better converting the JSONL to CSV and then import it to your database using the command line mysql/postgres tool instead of using NodeJS

2 Likes
3

Thanks Apple, is that a bit of a manual thing, because I am meant to refresh this data once a week, as well as do an incremental refresh every 24 hours.

4

You could write a Bash script and add it to cron or something

2 Likes
5

I see from your link that you already walked pretty far down this road yourself too.

6

Don't be afraid to speak with Digital Oceans support team Paul. They really are very good and extremely helpful in cases like this.

1 Like
7

hmm, will give it a try, got nothing to lose for sure, and maybe they have some solution.

1 Like
8

Have to say, i did warn you about node memory heap limits :grin:

1 Like
9

Have you considered runnng this as an script locally, filtering as you need to the small numbers you actually need then pushing the data to production from local using a second server connection?
27gb of data processed online in a droplet is always going to be problematic.

10

So quite an interesting development, if I turn off the output for 2 of my steps, and comment out all my debugging console logs, it no longer runs out of memory at 59 000 records, but actually manages the entire 2 585 753 records and it does it all in only 55 minutes and 12 seconds.

3 Likes
11

impressed

12

Yeah, pretty shocking, i have to admit, but so happy its working, just wish i knew how much memory it used to do it, so how i know how close I am to an issue, would hate the provider to add in 100 more lines next week and suddenly its too much.

13

you could manually append the file to itself to double the records and do a test?

could be fun

1 Like
14

True that, test a 56gb file, shame poor server will need therapy after that.

1 Like
15

This is forbidden in my book of life

Never load entire file contents of a big file into memory. You might need to rework this to only load X lines starting at Y offset, and adjust your server action for this, and repeatedly run the server action with increasing offsets (though I remind you this isn't the solution I'd personally do, my ideal solution was stated in previous posts)

1 Like
16

Not really viable as the schema is pretty complex and nested with multiple object arrays

I agree, had this conversation with him several times, i believe that is what he is doing now, last we spoke chunks of 100 records but i think he may be looking at tweaking that upwards

2 Likes
17

Yes, I am loading the main file of 28GB with a known byteStart and byteEnd of only 100 lines at a time, and then inserting the 100 records to the table before i go and fetch the next chunk of data.

Thanks for all the advice everyone, so happy this is finally working, and so much faster than I ever anticipated.

Strangely enough even the server load when i look at the Digital Ocean graphs are hardly being hit that hard.

18

Probably it's just using one core because NodeJS is single-threaded, so if you have 4 cores you probably see a load of 0.25 (1/4; which is 100% of 1 core in 4 available)

19

Ahh, that makes sense, thank you for the info, did not realise that, would that mean you never need more than a single vcpu on docker machines? or if 2 different tasks ran would it then use the extra CPU

20

The database server is another process, so it could use another core