How to improve this blocking code

Wappler Extensions
#1

I have a custom module that returns an array of keys ( I need all keys in local_keys that are not in current_keys.)

Both input arrays are ~150k records.

It works, but is blocking other requests during the 30 seconds it takes to run. I don’t care about the 30 seconds as this is run in the background, but I do care that it is blocking user http requests.

I can’t find any documentation on how to get array’s into custom modules, so I am stringifying the arrays, and then parsing them out, which is the root problem I believe, not the filter.

exports.keys_to_remove = async function (options) {

    let current_keys = JSON.parse(this.parseRequired(options.current_keys, 'string', 'parameter current_keys is required.'));
    let local_keys = JSON.parse(this.parseRequired(options.local_keys, 'string', 'parameter local_keys is required.'));
    return local_keys.filter(x => !current_keys.includes(x));
    

};

Anybody have ideas on a better way to do this?

Edit: The problem is NOT in the parsing of the arrays, it IS the filter. Although there must be a way to accept arrays.

#2

You could do the this in an child process, in NodeJS using fork and in browser using webworkers.

How To Launch Child Processes in Node.js | DigitalOcean

For the browser there is also a nice library js-coroutines for doing heavy processing and keeping the page responsive. Not sure if it also works in NodeJS.

1 Like
#3

Boom! :boom:

That’s what I needed Patrick. Forking works like a charm.

Really appreciate all the education you provide!

#4

Patrick, you are Forking amazing!

1 Like
#5

You need to pass them just like everything else.
Eg: A query step returns an array. You can send this to custom module with direct binding: {{query1}}.