Node custom module: installing and using "@hyperbytes/wappler-get-file-from-url-or-api"

1.0 Custom extension Tutorial

Building on the excellent initial work of @mebeingken & @jellederijke i have added additional functionality to deal with APIs which do not return a correct file extension.

Initially developed to deal with the qrserver.com API, this extension extends GetFileFromAPI, which allows for custom filenames, to also allow forcing of an extension such as .png etc where the API call does not return an extension correctly

This extension now returns both the filename and the web path to the file.

This extension can download all types of file however for the purpose of this tutorial we shall deal with images.

2.0 Installation

Go to Project Settings and select the Extensions tab

Select “+” to add Extension

image

enter the extension name, @hyperbytes/wappler-get-file-from-url-or-api

Enter the extension name and click “Save”

The extension should appear in the extensions screen

Click “Save” in the Project Options

image

The extension will be installed, the progress can be viewed in the Output wiindow

2.1 Activating the extension

To activate the extension it is essential to COMPLETELY quit Wappler, a simple restart form the tray icon is insufficient. If unsure how to do this then simply restart your computer.

3.0 Using the extension

Select Workflows

image

Create a new API, i call mine “getfile”


add an action

image

Select API Connector, you should see an icon “Get file from API”

Click to add this to your API action

The action will be added

3.1 The basic file download

Give the action a name, this is mandatory, i call it “myfile”

image

The save path is the path to which you want to save your file. This would typically be into your public folder, i select /public/assets/uploads

NOTE, the folder must already exist, this extension will not create the folder and if the folder is not present the action will return an error! If necessary add a File Management -> Create folder action before using this extension including with dynamic paths

image

I then enter the URL of an image from another site, in this case I choose an image from the Wappler website, “https://wappler.io/assets/img/features/ft-design-1.jpg” and check Output to allow the image name and web path to be returned from the API action

** NOTE the returned path will mirror the input path trailing slash, “/public/assets” returns “/assets”, “/public/assets/” returns “/assets/”

image

I run this directly from the APi action screen

You can see the output in the browser screen

{"myfile":{"file_name":"ft-design-1.jpg","path":"/assets/uploads"}}

The file has been downloaded from the URL and placed in the selected folder

3.2 Giving you file a custom name

Check the Allow Custom Filename checkbox

image

This reveals a Custom Filename input and a new checkbox “Allow Force Extension” which we will deal with later.

Enter a filename, i use “mynewfile”, DO NOT enter an extension!

Save the Action and again run from the action panel

We see from the output the file has been renamed to “mynewfile.jpg”

{"myfile":{"file_name":"mynewfile.jpg","path":"/assets/uploads"}}

3.2 Downloading from an API source

In this example I shall use api.qrserver.com to return a QR Code image pointing to an URL

I use the endpoint

https://api.qrserver.com/v1/create-qr-code/?size=150x150&data=https:wappler.io

to create a QR code pointing to wappler.io

I add this to the API action and remove the custom filename

and run the action for the action screen

I see an error (debug is currently on)

{"status":"500","code":"ENOENT","message":"ENOENT: no such file or directory, open 'E:\\webs\\getfile-tutorial\\public\\assets\\uploads\\?size=150x150&data=https:wappler.io'","stack":"Error: ENOENT: no such file or directory, open 'E:\\webs\\getfile-tutorial\\public\\assets\\uploads\\?size=150x150&data=https:wappler.io'"}

This is because the API does not return a properly formed filename

I add a custom filename, “myqrcode”

and run from the action screen

The action runs BUT we can see the extension is not an image extension as the API has returned an incorrect extension

{"myfile":{"file_name":"myqrcode.io","path":"/assets/uploads"}}

Ther file is downloaded with the incorrect extension

renaming the file to myqrcode.png shown the file is valid, just the extension is correct

3.3 Forcing a custom file extension

To correct this we use the “Allow Force Extension” option

image

which opens another input box, i enter “.png”

and save the action

I run it from the action screen

We see the desired filename is now returned

and the file is downloaded correctly

{"myfile":{"file_name":"myqrcode.png","path":"/assets/uploads"}}

3.4 Downloading PDFs

In this case I enter the URL of a dummy PDF stored on w3.org, https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf

We run the action and view the output

{"myfile":{"file_name":"dummy.pdf","path":"/assets/uploads"}}

and can view the file in the folder

4.0 Summary

This extension can be used to download pretty much any filetype.
Again, thanks to @mebeingken and @jellederijke for the donor code for this extension which I could not have made possible without.

8 Likes

Great work @Hyperbytes @mebeingken and @jellederijke - This is an excellent extension, very handy and versatile.

Very useful extension, thanks a lot :clap: :+1: @Hyperbytes, @mebeingken and @jellederijke

1 Like

Thanks so much @Hyperbytes for this extension! Making heavy use of it already :smiley:

Question: is there a way to check the response value of an API? I am using the Discogs API to load images and they are rate throttling. I was able to implement a proper solution for the other API calls but not for the images where I currently have a hard-coded delay to ensure I am not running into an 429-error.

I will take a look when i get a chance and see if i can add some sort of status response.

That would be fantastic :star_struck:

I need this ability on an older asp site and also on aspx site. I know I know, everyone is doing node etc. But because of security concerns have to stay with this due to hosting their own servers. I wonder how hard it would be to port this over for those frameworks? Wondering if chatgpt could make that change over easier? Or if its even posssible?

Hello @Hyperbytes! Fantastic work! Thanks!!

One question: I'm trying to download an AWS/Digital Ocean signed download PDF and it is not downloading.

The URL is like this one:

https://sgp1.digitaloceanspaces.com/webappdocs/project_docs/quotes/000cb21f-bacc-4b90-bdfc5e-d68c21605b34/dummy.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=7DDFEUAGDMNSAK2E4YHJW7%2F20241121%2Fsgp1%2Fs3%2Faws4_request&X-Amz-Date=20241121T181735Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=0f67e7af1fc57ed04045fa410efb2b580fc7310cdc60779344e28efdd23423191

This URL is not from my Bucket, it's from a third party app and it is modified for security reasons.
Any tip to make this download work?

Thanks!

@baub sorry, not touched asp for 20 years and never got to grips with .net; went the PHP direction.
I have found AI pretty good at translating though so may be worth trying that route.

@otavionestares

Not a service i am familiar with.
Any error messages?

Thanks for your reply!
Yes, there is a error message:

Error at GetFile: [Error: EIO: i/o error, open '/opt/node_app/public/user_upload/doc/project-quote-comparison_815d0a097d7201749fa87d14a70fe3a0/working-drawings/BuiltGrid_Contract_BGDCT1473779_ataya-paint-services_agile-designer-homes_29112022_1.pdf.pdf?X-Amz-Algorithm=AWeS4-HMAC-SHA256&X-Amz-Credential=7DEUA2GDMNSAK2E4YHJW7%2F20241121%2Fsgp1%2Fs3%2Faws4_request&X-Amz-Date=20241121T181735Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Am2z-Signature=0f67e7af1fc57ed04045fa410efb2bdd580fc7310cc60779344e28efdd23423191']

It is weird... looks like it is trying to open the link using the save location path!!

Can i see your settings you are using with the module (hide anything sensitive)

Not an expert here, but that doesn't look right

@franse hehe! Yes! I changed the file name in this post and other infos e.g (auth codes) due the confidencial infos in the document :smiley: Thanks anyway!

@Hyperbytes sure!!

Screenshot 2024-11-22 at 14.48.07

var_folder_name : '/public/user_upload/doc/'+qry_retrieve_project.project_name.trim().slugify()+'_'+qry_retrieve_project.token+'/working-drawings/'
This folder exists in my server, ok?

URL is the link: Below a file link without sensitive infos (link valid for 7 days)

https://webappdocs.sgp1.digitaloceanspaces.com/project_docs/support_document/001d09e3-c593-4daa-b82e-2dbbc849d7e1/Node%20custom%20module_%20installing%20and%20using%20_%40hyperbytes_wappler-get-file-from-url-or-api_%20-%20Wappler%20General%20_%20Need%20Help%20-%20Wappler%20Community.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=7DEUAGDMNSAK2E4YHJW7%2F20241122%2Fsgp1%2Fs3%2Faws4_request&X-Amz-Date=20241122T175358Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=995c99e4cc36ed792f1cbdfe12d58431fb18f165996a7cfffbf86129f7a1aca6

@Hyperbytes I'm pretty sure the problem comes from here: ?X-Amz
Probably trying to write a file with ? which is a non permitted character :thinking:

@franse these infos are part of the signature of the file, without any of them, the file will not open.
@Hyperbytes I get successfully get only the name of the file and forced the filename, but still no success with the download...

Yes, what I mean is that maybe, the extension is trying to write a file by downloading it with ? on it.

@franse ahhh!! yes! probably!

For example:

  server-connect:app Executing action step APIFileDownloadAction +0ms
  server-connect:app options: {
  server-connect:app   file_url: 'https://community.wappler.io/uploads/default/original/3X/3/4/349e082001d92a67fa7cfdf609fee8f0d6cffbe4.png',
  server-connect:app   local_path: '/public/assets/downloads'
  server-connect:app } +1ms
  server-connect:output name: {
  server-connect:output   file_name: '349e082001d92a67fa7cfdf609fee8f0d6cffbe4.png',
  server-connect:output   path: '/assets/downloads'
  server-connect:output } +0ms

But adding ?value=hello:

{
"status": "500",
"code": "ENOENT",
"message": "ENOENT: no such file or directory, open 'C:\\xampp\\htdocs\\getfilefromapi\\public\\assets\\downloads\\349e082001d92a67fa7cfdf609fee8f0d6cffbe4.png?value=hello'",
"stack": "Error: ENOENT: no such file or directory, open 'C:\\xampp\\htdocs\\getfilefromapi\\public\\assets\\downloads\\349e082001d92a67fa7cfdf609fee8f0d6cffbe4.png?value=hello'"
}
1 Like