Performance problem with my website, advice and help needed

[nickneustroev]

Hi, everyone!

I made a website with Wappler.
It works well with a low number of users but when numbers increase, the problems with server begin.
Maybe you can help me with some advice.
.

Here is details.

1. Website is a platform for online quizzes with real live broadcasting. This is not SaaS, this is just a tool for one local company.

2. There are 3-5 games that take place every week, 2-hours long each. Rest of the time there is nothing happening on the website.

3. Website is simple. For the regular user it is just one page, divided in half. On the left side it is broadcast. On the right side is the form for answers. You can check it out here https://riddler-testing.ga/games/test555
It is test server, so you can even login in any name and send form too.

Also, there are 3 pages for admins, where they can change game settings, see the list of teams playing and check the answers.

4. Broadcasting is just an iframe for another app/service Facecast. So my website has nothing to do with it and, I suppose, broadcasting doesn’t affect my server.

5. For hosting I have a VPS from the popular local provider. It has 6 CPU and 6 RAM. On top of that I installed regular Plesk Web Admin. NodeJS used as a backend platform. MySQL for the DB.

6. Basically it is a plain default setup.

Custom changes are at minimum. In Plesk I turned off proxy mode to stop using Apache in order to make Websockets work. And I changed ModSecurity to “Nginx (ModSecurity 3.0)” and changed Rule Set for Custom, manually downloaded Atomic, i think
Also I have installed some free basic extensions in Plesk: ImunifyAV, Traffic Monitor. Also I have installed Plesk 360 Monitoring on VPS.
And that’s it, I believe.

To be honest, I am absolutely not familiar with all this server stuff. So I did just the basic things to make the website up and running.

7. About the problem itself.
When there are about 200 participants playing the game, it goes well.

But recently we tried 500 and it was not good. Participants have seen the website and broadcast, but there were some lost forms. Also, admin pages was loading too long (couple of second) and sometimes page doesn’t opened after refresh, you have to refresh it again to load.

8. For this last troublesome game I have Plesk 360 Monitoring installed on the server. I have screenshots of a situation for that day.

9. Currently I’m trying to get assistance from tech support from my hosting provider. Maybe they will be able to sort this out, but I don’t know.
.

Considering all this and urgency of situation I want to ask some advice and help from community:

1. Does 500 concurrent users count as some kind of a “above typical” server load, so the problem is legit?
Or this is more like a regular load so that means there are some mistakes in my code or server/Plesk settings?

2. What would you suggest I must do in order to diagnose and solve the problem? And what questions should I ask to hosting support?

3. What is the Wappler limit for this kind of project? (how many thousands of concurrent participants/players?) Considering the possibility of a powerful hosting setup, if needed.

.
I would provide any additional information about the project and server if it helps.

Also if someone would be able to provide a paid revision/check up of my project and/or server/Plesk setup, I would be highly grateful.

[nickneustroev]

Almost forgot. When I checked my nginx errors log in Plesk, I saw a lot of errors, all similar to this:

2022/04/03 22:06:14 [error] 3262#0: *72249 connect() to unix:/tmp/passenger.VjdF24f/agents.s/core failed (11: Resource temporarily unavailable) while connecting to upstream, client: 109.252.208.2, server: sitename.ru, request: “GET /socket.io/?EIO=4&transport=websocket HTTP/1.1”, upstream: “passenger:unix:/tmp/passenger.VjdF24f/agents.s/core:”, host: “sitename.ru”

I don’t know if this error points to a specific problem or it just shows that there is some trouble.

Also I got some response from hosting support.
The have run wrk and have this results

wrk -t12 -c500 -d60s https://sitename.ru/" target="_blank">https://sitename.ru/
Running 1m test @ https://sitename.ru/
12 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 272.44ms 143.34ms 1.32s 81.44%
Req/Sec 119.94 59.48 470.00 67.54%
83672 requests in 1.00m, 108.66MB read
Socket errors: connect 4209, read 1560, write 0, timeout 2206
Non-2xx or 3xx responses: 73479
Requests/sec: 1392.79
Transfer/sec: 1.81MB

And they saw same errors in nginx log.

They suggest make changes in nginx config according this topic:

How to configure Nginx with Passenger to process 500 requests/sec

[patrick]

It seems that CPU usage is very low, you can configure nginx/passenger to run multiple instances of the application, it will function as a load balancer and it should speed up your server a lot.

https://www.phusionpassenger.com/library/config/nginx/reference/#performance-tuning

Also when you use multiple instances you should use something like Redis for your session management and you can also use it to cache specific server routes from which you know the content doesn't change much.

Content Caching for Node.js with Redis

Intro Now you can empower Redis caching service with Docker and select which Node.js pages and API Actions to be cached. Then instead of processing them fully on each request, they will be cached in the super fast Redis memory cache and served so much faster to the clients! This is the perfect Server Side caching for all your web visitors! This will increase your site speed at least 10 times! Note: All this is possible only with Node.js and Docker and it is for now only under the Experimenta…

[nickneustroev]

Patrick, Hi!
Thank you for the feedback.
Understood, we will try to configure nginx/passenger.

[patrick]

Alternative it is also possible to make some small changes to the js code to have a cluster.

https://nodejs.org/dist/latest-v16.x/docs/api/cluster.html

Important to know is that memory is not shared between the processes, so for things like sessions you should not use memory storage and also for your sockets I would advice using Redis.

[Apple]

Patrick pretty much summed the problem - NodeJS is a single-threaded app, so you need to run multiple instances in parallel to take advantage of a multi-core processor

https://www.phusionpassenger.com/library/config/nginx/reference/#passenger_min_instances

I’d recommend doing this at Passenger/nginx config rather than using NodeJS’s clustering, primarily because you’d be deviating from standard Wappler code that could be overridden with an update.

And just to elaborate a bit more, looking at the CPU graph, you see there’s a tendency to form an upper-bounded line at 20% CPU usage - this implies one CPU core is at 100%, while the rest are unused. 100% / 6 cores = 20% / core

On the SSH terminal you can use the command “top” to view CPU usage per-process. You could also see that way how many “node” (NodeJS) processes are running to confirm you have multiple instances running in parallel (or you can run “pgrep node” as well)

Good luck!

P.S.: I’m busy, but if you truly get stuck feel free to reach out for a paid consultation

[nickneustroev]

Thank you a lot for the addition!
I definitely see the situation more clearly now.

Seems like the problem is narrowed down, so I hope the hosting support will be able to find the solution.

[nickneustroev]

@patrick @Apple I almost forgot to write a follow-up.

So I sent your thoughts and links to my hosting support and they managed to set up multiple instances for nodejs.
And the website really began working smoothly with the ability to hold much more users then before.

So again, big thanks for the help, guys! You really saved the day back then.

Although, indeed, this caused some problems with sockets. It wasn’t critical for the project though. I’m about to solve this too.

[patrick]

To solve the sockets and session problem I suggest to use Redis which we support directly from Wappler, there are other ways like using database or the filesystem to store the data that needs to be shared between the processes.

[nickneustroev]

So, let’s imagine I have:

• nginx is configured on the server to support multiple NodeJS instances,
• Docker Hosting for hosting and NodeJS for server model,
• build-in Redis enabled and it enabled for required server actions and pages.

It would already work alright?
Or should I make additional manual settings for Redis?

[patrick]

In the Server Connect Settings enable Redis in the Redis tab and in the Sessions tab set the Session Store to redis. With redis enabled you can also use the Cache Time property on routes that may be cached. Sockets will automatically use redis when it is enabled.

[nickneustroev]

Got it, thanks! So basically everything preinstalled out-of-the box, cool.

By the way, regardless of this topic overall – is it technically possible in the future to make Wappler itself to set multiple instances web apps in order to take advantage of a multi-core processor? Without help of hosting support or any other kind of professional assistance.

[George]

@nickneustroev

is it technically possible in the future to make Wappler itself to set multiple instances web apps in order to take advantage of a multi-core processor?

Do you mean the Wappler app or your nodejs site?

[nickneustroev]

I mean nodejs site

[patrick]

We can make multiple instances easily, but most hosting already offer this. It is better to use a good process manager like Pasenger which is used by your host or something like pm2. The advantage of such process manager is that it runs separately from the node app and it handles crashes, large memory use and also logging.

Also as you know there are some extra issues with running multiple instances, like sessions and sockets must use some persistence storage to share there data between processes. This is easy with Redis, but not everyone has Redis.

If there are more requests for multiple instances we will perhaps include it. It is easy to implement with the cluster api. Cluster | Node.js v16.14.2 Documentation (nodejs.org)

[nickneustroev]

@patrick

If there are more requests for multiple instances we will perhaps include it.

Thanks! Understand you, will see.

But what do you think about this quote below? Basically, it is not the best way to use pm2 or Cluster, because changes can be altered after some update?

@Apple

I’d recommend doing this at Passenger/nginx config rather than using NodeJS’s clustering, primarily because you’d be deviating from standard Wappler code that could be overridden with an update.

[Mr.Rubi]

@patrick

If there are more requests for multiple instances we will perhaps include it.

That would be very appropriate. Some of my projects use private servers and sometimes there are performance problems. Nothing critical yet, but if the user load increases, it will become critical. To be able to configure the cluster directly from the Wappler would be just perfect.

[Apple]

@nickneustroev

But what do you think about this quote below? Basically, it is not the best way to use pm2 or Cluster, because changes can be altered after some update?

pm2 is fine, I was referring to a built-in NodeJS clustering functionality

Didn't even cross my mind to use something like pm2 in front, I think that's an acceptable solution

[nickneustroev]

@patrick @Apple So, I decided to try to manage the project using Docker Hosting instead of Plesk I’m currently using. It will cut off the websockets problem.

The question is, how to correctly configure multiple instances in that case.
If I’m gonna use Docker, then it must be settled somewhere in the project itself, right?

Considering I have no experience in administering Docker, nodejs or whatever, do you think I would be able to configure it myself following instructions? (via clustering or pm2 or any other way) Is it a complicated process?