How to upload files from local Windows UNC path (outside project root) to S3

Wappler General Need Help
1

Hi,
I have files on a Windows Server which I want to upload by cronjob to my Managed Server (nodejs) or S3 bucket.
The path of the file is in a database.
My thinking was to use my local NodeJS Webserver which is in the same local Network.

I found the following information on stackoverflow:

What should I think about if I try to implement this with Wappler?

Should this work?

At the moment I use a script with ftps for uploading the files. It does work but I would like to optimize the workflow.

2

@patrick Would it be possible to create a custom module for download from a path outside root for this case? With this I could download to a temp folder and upload to the server. Or do you have another idea how this could be accomplished?

3

You could copy the file from your network drive to a temp folder to upload it from there to S3, but if you are already making a custom module, then upload directly to S3.

Copy action:

const fs = require('fs-extra');
const { toSystemPath } = require('../../../lib/core/path');

exports.copyFromNetwork = function(options) {
  options = this.parse(options);
  return fs.copyFile(options.from, toSystemPath(options.to || '/tmp'));
}

For the from option use the full network path, the to option is a Wappler path and defaults to /tmp.

1 Like
4

This looks perfect. I will try this.

5

Hi @patrick

I ran into an issue. I got the error ā€œError: EPERM: operation not permitted, copyfileā€.

I found the following information on github:

I could solve it by giving a filename in the to option. How can I combine ā€˜/tmpā€™ with a file from the options?

6

@patrick

In addition to the copy action I tried the direct upload to S3.

I used the following code:

const fs = require('fs-extra');
const { toSystemPath } = require('../../../lib/core/path');

exports.copyFromNetworkToS3 = async function(options) {
    const provider = this.parseRequired(options.provider, 'string', 's3.uploadFile: provider is required.');
    const Bucket = this.parseRequired(options.bucket, 'string', 's3.uploadFile: bucket is required.');
    const path = toSystemPath(this.parseRequired(options.path, 'string', 's3.uploadFile: path is required.'));
    const Key = this.parseRequired(options.key, 'string', 's3.uploadFile: key is required.');
    const ContentType = mime.lookup(Key) || 'application/octet-stream';
    const ACL = this.parseOptional(options.acl, 'string', null);
    const s3 = this.getS3Provider(provider);

    if (!s3) throw new Error(`S3 provider "${provider}" doesn't exist.`);

    let Body = await fs.readFile(path);

    return s3.putObject({ Bucket, ACL, Key, ContentType, Body }).promise();
}

Is this correct?

My problem now is to create the hjson file with the droplist for selecting the s3 provider.

I assume it should be something like this:

{ name: 'provider', optionName: 'provider', title: 'Provider', type: 'droplist', value: ???, required: true, defaultValue: ''},

What do I have to put in the value field for the droplist?

7

The code looks good. There is currently not a way to get a list of s3 providers, so you should probably make that a simple text input.

It seems there is a special upload method in the aws sdk that can work with streams. I will be updating our component for that.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property

Using a stream would use less memory then reading the full file first and them upload the content.

1 Like
8

Would using the stream also work with other s3 providers than aws @patrick ? Iā€™m using ionos because I have to use a german providerā€¦

9

The new update for S3 will be included in todays Wappler update, here the updated module for if you want to test it out. Streams will also work on other s3 providers.

s3.zip (1.0 KB)

1 Like
10

This is pretty awesome! I got it to work!

This is the final code of the js file:

const fs = require('fs-extra');
const mime = require('mime-types');

exports.copyFromNetworkToS3 = async function (options) {
    const provider = this.parseRequired(options.provider, 'string', 's3.uploadFile: provider is required.');
    const Bucket = this.parseRequired(options.bucket, 'string', 's3.uploadFile: bucket is required.');
    const path = this.parseRequired(options.path, 'string', 's3.uploadFile: path is required.');
    const Key = this.parseRequired(options.key, 'string', 's3.uploadFile: key is required.');
    const ContentType = mime.lookup(Key) || 'application/octet-stream';
    const ACL = this.parseOptional(options.acl, 'string', null);
    const s3 = this.getS3Provider(provider);

    if (!s3) throw new Error(`S3 provider "${provider}" doesn't exist.`);

    let Body = await fs.readFile(path);

    return s3.putObject({ Bucket, ACL, Key, ContentType, Body }).promise();
}

This is my current hjson file:

{
  type: 'copyFromNetworkToS3',
  module : 'copyFromNetworkToS3',
  action : 'copyFromNetworkToS3',
  groupTitle : 'My Modules',
  groupIcon : 'fas fa-lg fa-project-diagram comp-images',
  title : 'Copy file to S3',
  icon : 'fas fa-lg fa-copy comp-images',
  dataPickObject: true,
  properties : [
    {
      group: 'Copy file to S3',
      variables: [
        { name: 'provider', optionName: 'provider', title: 'Provider', type: 'text', required: true, defaultValue: ''},
        { name: 'name', optionName: 'name', title: 'Name', type: 'text', required: true, defaultValue: 'transferFile', help: 'help'},
        { name: 'bucket', optionName: 'bucket', title: 'Bucket', type: 'text', required: true, defaultValue: '', serverDataBindings: true,
          help: 'help'},
        { name: 'key', optionName: 'key', title: 'Key', type: 'text', required: true, defaultValue: '', serverDataBindings: true,
          help: 'help'},
        { name: 'path', optionName: 'path', title: 'Path', type: 'text', required: true, defaultValue: '', serverDataBindings: true,
          help: 'help'},
        { name: 'acl', optionName: 'acl', title: 'Access', 
            type: 'droplist', 
            values: [
            {title: 'Private', value: 'private'},
            {title: 'Public Read', value: 'public-read'},
            ], defaultValue: '',
            help: 'help'
        ,}
        { name: 'output', optionName: 'output', title: 'Output', type: 'boolean', defaultValue: false }
      ]
    }
  ]
}

I will make the module available for download as soon as I get the hjson file ready.

Thank you for your help @patrick

1 Like