[NodeJS] Database Bulk Insert
I’m publishing here a custom extension I made a few months ago to bulk insert data into a database. It kinda works but I think CSV import is broken. I spent many hours on this and then I got annoyed and gave up. I don’t have the time or motivation to maintain this extension any further. I was secretly hoping the Wappler team would release something official, but I guess it’s taking too long, so I’m releasing it here in case it’s helpful to someone. If you have the knowledge you’re free to fix bugs and maintain it.
This extension is how I envisioned a high-performance bulk insert step for Wappler. It allows selecting the input from either an array of objects or a CSV/TSV file (currently broken) and perform a myriad of on-the-fly changes, such as renaming columns from the input data, so you can precisely map certain columns from the input to certain columns to the output (“Renamed columns”), and even omit (“Allowed columns”).
In addition to adding rows, it also supports updating rows (“On conflict column name”, “Merge column name”).
The fields above support both string and array formats, meaning your expression can either be a string or an array, depending on your preference.
Best practices were attempted, such as support for chunking data (“Chunk size”) to avoid high RAM usage. Unfortunately, such practices were deemed a failure, as the script still crashes with large datasets due to RAM exhaustion.
“Allow user abort” is an experimental feature to abort adding rows if the client closes the HTTP connection (i.e. closes the page), however, it relies on deprecated NodeJS functionality (and the successor doesn’t really work great).
bulk_insert.zip (3.4 KB)
Unzip it to extensions/server_connect/modules. If the folder doesn’t exist, create