Timing Issue With File Copy & Folder Zip

Wappler General Bugs
#1

Wappler Version : 3.9.9
Operating System : W10
Server Model: NodeJS
Hosting Type: Docker

Expected behavior

Looping through a bunch of files inside a folder, & copying them to another folder.
Then, creating a ZIP from the newly created folder.

Actual behavior

The newly created ZIP does not have the last two images, consistently, even though the image does get copied to the new folder.

How to reproduce

Here’s what my setup looks like (simplified for bug report):

  1. Create a new folder.
  2. Folder list step to get all files inside an existing folder.
  3. Use repeat to loop through each image separately to identify which images I need to duplicate.
  4. Use FILE COPY step when condition to duplicate is true.
  5. Outside the repeat, I use Folder ZIP step to create ZIP of the folder created in step 1.

In output, I see all images but the last two, consistently.
So, I tried adding a delay using WAIT of 1500 ms, and now I see all images. This is not at all workable, since there is no guarantee that last 2 images will get copied in 1.5 seconds.

Are the COPY & ZIP steps not sequential?
Shouldn’t the COPY step complete only after the image is actually copied?
Please help.

Wappler 4.0.0 Beta 10 Released
#2

The script should indeed wait for the files to be copied. In NodeJS we use async functions and each step waits until the previous step resolves, also the repeater waits until the steps within have finished before going to the next entry.

#3

So what could be the issue here?
I have never seen such an issue since the steps are designed to run sequentially in SA.
Becuase without the wait, ZIP step is getting executed before COPY steps inside the repeat have even finished.

#4

I rechecked all our code and we have a wait on every location. I think it is more a problem on a higher level with NodeJS or the OS. Maybe the copy operation returns before the whole file is written because of HDD buffering or so.

Checking the NodeJS docs: https://nodejs.org/dist/latest-v14.x/docs/api/fs.html#fs_fs_copyfile_src_dest_mode_callback

It says there Node.js makes no guarantees about the atomicity of the copy operation.. Not sure if that means that they can’t guarantee that the copy operation completely finished when the callback is called.

#5

I think that means that if node crashes or the copy is interrupted you will end up with a corrupt file. It doesn’t ensure atomicity when copying.

#6

So should I just put in a WHILE loop on ‘if last copied file exists’ & generate zip when this loop breaks?
Because waiting 1500ms or any other random number would not work.
I can also put in a limit of say 15/20 iterations in WHILE to ensure it doesn’t go into infinite loop.

Or if you have any other ideas… @JonL?

#7

Unfortunately I don’t have any suggestions @sid. I haven’t used that module yet so I haven’t encountered the issue you mention.

#8

I have to do some research on why it happens, but maybe having an action like waitForFile would be handy. You can experiment with a while loop, but not sure if that is an ideal situation.

#9

Thanks Patrick. Let me know if you make some progress in this.
Will setup the while loop logic. Its better than wait at least, for now.

#10

I try to reproduce your problem, but for me it zips all files.

My steps:
image

#11

Maybe this will clarify my setup a bit more:
I am using Docker & NodeJS.
My list of paths come from a DB.
Then I create a new MAIN folder.
Then in repat, I check if file exists. If yes, then I copy that file in the new MAIN folder, but in a separate folder of itself (with the help of CREATE FOLDER option).
Finally, after repeat, I have the ZIP option.

In the ZIP of the MAIN folder, I expect about 7 sub-folders, with image in each of them. Count of images range from 2-7 in each sub-folder.
Instead, what I get is the last 2 sub-folders are blank in the ZIP.

But, if I add a delay of 1500ms, those 2 also get ZIPped with images.

#12

Hi @patrick,
I am seeing this more often now. I have two more SAs where similar looped file copy & zip setup is there.
And the ZIP generated is missing sometimes few or all files.

#13

I’ve updated the zip module to throw an error if anything goes wrong, normally it would skip warnings like when files do not exist.

zip.zip (1.1 KB)

Also try to do some debugging, like adding as exists step after the copy to see if it exists on the new location. Turn on all the output to see the results in the returned json.

#14

image

I do have a similar setup. This variable lastFile is declared outside the repeat, and after repeat, I check if variable has a value, and in then part - I check using while loop if that copied last file exists yet or not.

Unfortunately, the value does not always get set timely, so empty ZIP gets generated.
What I assume is happenning here is that while the file copy is still running, repeat goes to the next iteration.
This keeps hapenning and when it completes, the variable is actually empty at the time of condition step.

This goes completely against the sequential async flow of server actions, so not sure why its breaking here. Will test out the updated file and report back.

#15

Ran the SA with updated ZIP and I see numerous logs about file but all of them are after the SA has completed.
Usually, SA execution breaks at a step, or prints the warning just below the step in logs.

Have shared the logs with you privately.

#16

Could you do the exists step after the file copy and check if the file on the destination exists. I want to know if the copy step continues before the actual file is completely copied.

#17

Shared.

#18

Update FS lib file as shared by @patrick to fix this issue in NodeJS.
fs.zip (1.6 KB)

1 Like
#19
#20