Form Data Content Size Limit

Wappler General Bugs
1

Wappler Version : 5.5.3
Operating System : W11
Server Model: NodeJS

When sending huge amount of text data from client side over a POST form submission, the data received on server is truncated.
I have a repeat, where I am sending 331 items, each being a JSON object having two big URLs and a few ids in key-value pairs.
On server, I can just see 239 items in the $_POST object, where 239th item itself is also incomplete.

I tried searching for increasing size of POST FORM data in NodeJS Express, but suggested solution did not work.
I can send big files in POST FORM, but when sending just plain text, I can see this issue.

This is all happening on local NodeJS server, so not a remote server configuration issue.

2

The form data is being parsed in express by middleware. Depending on the content type it uses different parsing middleware.

For form post of type application/x-www-form-urlencoded it uses the urlencoded middleware. Default parameter limit is there 1000. We set the options extended to true which causes the middleware to use the qs library which also has some limits.

For form post of type multipart/form-data is uses express-fileupload middleware.

None of these have a limit of 239, so not sure which limit you are reaching.

3

Thanks for replying Patrick.

The number 239 is just the count of array items that I am receiving. I send 311 from client side, but received only 239 on server side.

I am not sure what 1000 means.
It it means 1000 keys, I am sending about 1250 key-value pairs in my urlencoded form submission.
How/where do I increase this number?

I could find qs reference in /lib/setup/upload.js, but I am not sure if this is used for urlencoded requests.
I don’t have the exact use case anymore, so I can change and test immediately - will have to make the test case first.
But is this the place to change the arrayLimit?

Also, any reason why the limit is set to 1000 by default - by qs and express both? Is it security related or just a precautionary limit?

4

It is for security to prevent users from creating large object or arrays on your server which could even crash your server. Depending on the content-type, for multipart/form-data you need to edit lib/setup/upload.js and for application/x-www-form-urlencoded you need to edit lib/server.js.

From the qs library documentation

By default, when nesting objects qs will only parse up to 5 children deep. This means if you attempt to parse a string like
'a[b][c][d][e][f][g][h][i]=j' your resulting object will be:

var expected = {
    a: {
        b: {
            c: {
                d: {
                    e: {
                        f: {
                            '[g][h][i]': 'j'
                        }
                    }
                }
            }
        }
    }
};
var string = 'a[b][c][d][e][f][g][h][i]=j';
assert.deepEqual(qs.parse(string), expected);

This depth can be overridden by passing a depth option to qs.parse(string, [options]):

var deep = qs.parse('a[b][c][d][e][f][g][h][i]=j', { depth: 1 });
assert.deepEqual(deep, { a: { b: { '[c][d][e][f][g][h][i]': 'j' } } });

The depth limit helps mitigate abuse when qs is used to parse user input, and it is recommended to keep it a reasonably small number.

For similar reasons, by default qs will only parse up to 1000 parameters. This can be overridden by passing a parameterLimit option:

var limited = qs.parse('a=b&c=d', { parameterLimit: 1 });
assert.deepEqual(limited, { a: 'b' });

qs will also limit specifying indices in an array to a maximum index of 20 . Any array members with an index of greater than 20 will instead be converted to an object with the index as the key. This is needed to handle cases when someone sent, for example, a[999999999] and it will take significant time to iterate over this huge array.

1 Like
5

Thanks for the explanation. Makes sense.

I checked the server.js file, and I see that qs reference is not added to this page. Comparing with upload.js, I assume the verify function in server.js will have to modified?

image

6

It is the express.urlencoded that needs to be modified, you can increase the parameterLimit for it.