How to Insert Bulk Data into Database While Checking for Duplicates in Wappler

Wappler General Need Help
1

Hi everyone,

I'm working on a Wappler project where I need to insert a large batch of records into a database (MySQL in my case). However, I want to make sure that duplicate entries (based on a unique field like email or user_id) are not inserted.

Here's what I'm trying to achieve:

  • Upload or provide a bulk dataset (e.g., via CSV or a JSON array).
  • Loop through the dataset in Wappler.
  • Before inserting each record, check if it already exists in the database based on a unique key.
  • If it doesn't exist, insert the new record.
  • If it does exist, skip or optionally update it.

So far, I'm using Server Connect and repeat steps to loop over the dataset, but I'm not sure how to efficiently:

  1. Check for duplicates inside the loop.
  2. Prevent duplicates without making the process slow or heavy.
  3. Handle bulk operations in a performant way.

Questions:

  • What's the best approach in Wappler to handle this?
  • Should I use conditional queries inside the repeat step, or is there a better native feature or module in Wappler to manage this?
  • Are there performance best practices or recommended patterns for large imports with duplicate checks?

Any tutorials, step-by-step guides, or examples would be greatly appreciated!

Thanks in advance :pray:

2

Hi,

Unfortunately, the most efficient approach is not implemented in Wappler, it's by using INSERT IGNORE (or ON CONFLICT)

I've created a feature request if you can find the topic

In fact, I've also created a custom extension for NodeJS if you have interest, you can search the forum

You can still show what you've done to see if it can be optimized, but unless someone comes with a creative trick it's still going to be slow compared to INSERT IGNORE

1 Like
3

This is how i did it.
Hope it helps you.. ... but i guess it all depends on 1 or 100 0000 records :slight_smile: think it will all depend on the amount of data... so you will have to test to see how efficient this might be.

Upload the csv data.. check if the "item" is already in the database.. if so.. skip it .. else... insert ... once done.. delete the csv file from the server...

loop insert
loop insert770×526 17 KB