Best way to download remote images to local filesystem in Node.js?

Hi Wapplers,

I need to download images from various external URLs to my local server storage within a Node.js project.

My Requirements:

  1. Download: Fetch an image from a remote URL.

  2. Save & Rename: Save the file to a specific local folder (e.g., /public/assets/images/) and give it a custom name.

  3. Reliability: Some external servers block simple requests.

My Questions:

  • Is the "File Download" component the best choice for this, or should I use an "API Action" and save the response as a file?

  • How can I pass a User-Agent header to the request to avoid being blocked (403 errors) by remote servers?

  • Are there specific settings in Node.js to ensure the download doesn't hang if a remote server is slow?

Setup:

  • Wappler: Latest

  • Server: Node.js

Thanks for any tips or best practices!

Sounds like you'll need a smart file retrieve utility since you'll come across various server responses (or obstacles). The server connect's File Download component, in my experience, is for allowing a user to download a file to the browser from the server, not to retrieve a remote file. I don't believe that the File Download component will serve your needs. API Action allows you to interact with remote API's, so this will help you only when the remote image can be retrieved by a remote API. If the image is just sitting in a public folder somewhere, the API Action won't be any help.

If the number of images to be retrieved and the rate of retrieval are both modest/reasonable, a front-end solution may work best for you. A user could enter in a URL, or a list of URL's, and Javascript could easily retrieve the image files, include error handling, sending HTTP headers, etc., then POST them to your own API on your local server for saving.

Take a look to this extension:

1 Like

Thanks for the tip!

Unfortunately, it doesn't work for me.

Even when I specify the image path directly, the images don't receive any data.

Can you show me how you are calling the extension.
If should work,its a well established and proven extension.
Are any errors thrown?

Actually, on looking that extension interface looks all wrong, sure you are using the correct extension and the latest version?

It should have a "Allow Custom filename" option if correct latest version

This is the one you need.

npm is : @hyperbytes/wappler-get-file-from-url-or-api

I'm not sure if I installed the extension correctly.

My first attempt failed, so I manually placed the two files in /server_connect/modules.

It then appeared in API Connector, but it didn't work.
The "Allow Custom filename" option wasn't even displayed.

I deleted the files again and tried installing it via extensions. It's now listed there. However, even though it's shown in extensions, it's not listed in modules.

I think I've done something fundamentally wrong.

Check the project updater modal, maybe something needs to be updated.

Also, sometimes Wappler needs a restart when installing extensions

If you did not use the installer there is a likelihood that the dependencies of axios, uuid and fs-extra are not installed.
I suggest you remove the files you manually added and install it the correct way via the project.

Then click save to install the extension.

I have just tested it and it installed correctly

I've tried exactly that several times, but unfortunately it doesn't work:

Here's the error message I'm getting.

When I disable my firewall, it seems to work; at least there's no error message.

However, I can't find the extension in the API connector.

Don't know if it's related, but your path has spaces in it, try to change it

This seems to be a permission issue within your node_modules folder.
The error is being thrown by the installer rather than the extension.

It seems to be trying to remove the @pkgijs/parseargs/examples folder which seems to be locked for some reason.
I have no idea why Wappler would be trying to do this, that is not related to anything in the actual extension.
As this is the installer throwing the error I cannot give a definitive answer as to why,
Perhaps @George could assist?

Manual Install
Unzip these 2 files and add to the /extensions/server_connect/modules folder of your project.

FileDownloadModule.zip (2.2 KB)

Now manually add the dependencies.
Go to Wappler terminal add/update them manually like this:

npm install axios

This will fetch the axios module which is used by the extension to fetch the file and is not installed by Wappler as standard

This will probably be sufficient (after restart) however is still not working try the below additional npm installs

npm install uuid

(uuid is normally installed as standard but this ensure it is installed and up to date)

npm install fs-extra

(fs-extra is normally installed as standard but this ensure it is installed and up to date)

Do a full Wappler restart
The extension should then work.

Also, seems the path is too long..
Windows has a limit for this

I had my project folder in a Google Drive folder.

Now I've moved it directly to my desktop.

The installation seems to have worked.

I've tried different images.

Unfortunately, I'm getting this error message online.

{status: "500", code: "MODULE_NOT_FOUND",…}
code
: 
"MODULE_NOT_FOUND"
message
: 
"Cannot find module 'axios'\nRequire stack:\n- /home/cgevents/domains/cg-events.de/public_html/app/extensions/server_connect/modules/FileDownloadModule.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/core/app.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/core/middleware.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/setup/routes.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/server.js\n- /home/cgevents/domains/cg-events.de/public_html/app/index.js\n- /usr/local/lsws/fcgi-bin/lsnode.js"
stack
: 
"Error: Cannot find module 'axios'\nRequire stack:\n- /home/cgevents/domains/cg-events.de/public_html/app/extensions/server_connect/modules/FileDownloadModule.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/core/app.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/core/middleware.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/setup/routes.js\n- /home/cgevents/domains/cg-events.de/public_html/app/lib/server.js\n- /home/cgevents/domains/cg-events.de/public_html/app/index.js\n- /usr/local/lsws/fcgi-bin/lsnode.js\n    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)\n    at Module._load (node:internal/modules/cjs/loader:981:27)\n    at Module.require (node:internal/modules/cjs/loader:1231:19)\n    at require (node:internal/modules/helpers:177:18)\n    at exports.APIFileDownloadAction (/home/cgevents/domains/cg-events.de/public_html/app/extensions/server_connect/modules/FileDownloadModule.js:6:19)\n    at App._exec (/home/cgevents/domains/cg-events.de/public_html/app/lib/core/app.js:757:49)\n    at App.exec (/home/cgevents/domains/cg-events.de/public_html/app/lib/core/app.js:682:16)\n    at App.define (/home/cgevents/domains/cg-events.de/public_html/app/lib/core/app.js:664:16)"
status
: 
"500"

This error message appears locally.

{status: "500", message: "Download failed with status code 404",…}
message
: 
"Download failed with status code 404"
stack
: 
"Error: Download failed with status code 404\n    at C:\\Users\\chris\\Desktop\\web\\relaunche\\extensions\\server_connect\\modules\\FileDownloadModule.js:40:31\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
status
: 
"500"

axios is one of the required dependencies, the error indicates it is missing.

Check it is installed locally, you should see it in the node_modules folder.

Normally the installer adds all dependencies for you but if amnually instlaled the4y need to be added manually.
If missing, you forgot to run

npm install axios

in the Wappler terminal

Does it work locally or is the error only on your production server.
Some servers require the dependencies to be manually updated from a control panel.

what server technology are you using, docker or VPS?

P.S. storing projects on cloud services is know to cause issues, i recommend you don't do it.

Hmm, everything seems to be okay here.

Questions unanswered.

  1. does it work from localhost or is error just in production?
  2. What server technology are you using? Docker or vps etc
  3. If vps, what control panel i.e plesk, cPanel