Panel will slow down drastically if one node is down

[OpenSource03]

Current Behavior

If one of nodes go offline, the panel will slow down. Additionally, it'll start throwing 502 bad gateway errors every now and than.

Expected Behavior

Once panel determinates that a node is down, it should no longer try to reach it causing consistent website errors and poor performance.

Steps to Reproduce

1. Take one wings off.
2. Preferably completely turn off the machine it's located on or disconnect it from the internet source
3. Try using the panel

(I’m using CloudFlare)

Panel Version

1.7.0

Wings Version

1.6.0

Games and/or Eggs Affected

No response

Docker Image

No response

Error Logs

https://bin.ptdl.co/q1jj5/

Is there an existing issue for this?

• I have searched the existing issues before opening this issue.
• I have provided all relevant details, including the specific game and Docker images I am using if this issue is related to running a server.
• I have checked in the Discord server and believe this is a bug with the software, and not a configuration issue with my specific system.

[camwhi-te]

Are you using Cloudflare on the Panel? Could be an explanation for the 502 errors you're receiving.

[iamkubi]

This is easy to reproduce. If I have a panel running (behind cloudflare) with 1 node up and 1 node down I will get 504 errors on 50% of refreshes hitting any panel URL.

I personally experienced this on 0.7 but have not tried to replicate it on 1.x. Based on reports in Discord (and here) I'm fairly sure it still exists.

[OpenSource03]

Are you using Cloudflare on the Panel? Could be an explanation for the 502 errors you're receiving.

I am, forgot to mention. I updated my initial comment.

[DaneEveritt]

Are you sure this isn't Cloudflare just assuming it is down because of the 504s? I explicitly return a 504 from the API endpoints that reach out to Wings, so if Cloudflare is seeing that and calling the whole site down as a result, thats an issue with Cloudflare.

504 Gateway Timeout
The server, while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request.

[OpenSource03]

Are you sure this isn't Cloudflare just assuming it is down because of the 504s? I explicitly return a 504 from the API endpoints that reach out to Wings, so if Cloudflare is seeing that and calling the whole site down as a result, thats an issue with Cloudflare.

504 Gateway Timeout
The server, while acting as a gateway or proxy, did not receive a timely response from an upstream server it needed to access in order to complete the request.

You can't just say "it's an issue with CloudFlare". Cloudflare is used by millions of websites, and just moving forward like it was nothing is not a great option. On top of periodic error pages, the panel can also freeze or get extremely slow, some servers may continue to load forever, even if not on that node etc etc etc.

I'm sure there can be a solution to this issue that would work with CloudFlare and any other CDN whatsoever.

[mrxbox98]

This might be related to 4964d29. This was a commit from 0.7 though so I do not know if the code is still present in the latest version.

[OpenSource03]

I'd say it's somewhere in

/var/www/pterodactyl/app/Repositories/Wings/DaemonServerRepository.php (suspension function)

Likely due to undefined timeout in GuzzleHttp Client. Adding a timeout did help a little, but more work needs to be done. This, of course, only made the panel a little smoother while a node was down.

[DaneEveritt]

Can you expand on what you mean about an undefined timeout, they're set here: https://github.com/pterodactyl/panel/blob/develop/app/Repositories/Wings/DaemonRepository.php#L72-L73

Can you please also include far more details about "panel slows down", because there isn't much I can do to debug that sentence or guide you towards better resources/ideas to try.

[OpenSource03]

Can you expand on what you mean about an undefined timeout, they're set here: https://github.com/pterodactyl/panel/blob/develop/app/Repositories/Wings/DaemonRepository.php#L72-L73

Oh, didn't see that, but it explains the issue. The default timeout is 30 from config. I had success even with 1.5 but didn't test it in production. Maybe it could produce issues via such a small timeout. I've implemented timeout of 1.5 in DaemonServerRepository suspend function but didn't know that there is one in getHttpClient(). Nonetheless, it made the panel a little smoother in case of a node being down, but it was still slower, unresponsive, and with periodic lovely 502 errors popping up.

Can you please also include far more details about "panel slows down", because there isn't much I can do to debug that sentence or guide you towards better resources/ideas to try.

Alright so here's the video: https://youtu.be/YEnfterMMx8

• At 0:26 we've stopped Wings.
• At the same time, we applied a rule iptables -A INPUT -p tcp --dport 8080 -j DROP. This is to simulate that the system is completely down. Should the host system be reachable, we'll only get 502, and no slowdowns.
• At 0:37 I've attempted a refresh on the page and experienced an extraordinarily long wait until it eventually loaded
• I attempted that a few more times after, same result

[DaneEveritt]

I've tried to reproduce this and cannot, I really don't know what to tell you here.

I have my entire node offline, and everything continues working exactly as I expect it should, minus the /resources API calls that return the expected 504 errors from the API but that is simply a front-end issue.

I believe this might be an issue with your php-fpm config. I can reproduce similar results only when flooding the API endpoint with a ton of requests at once (100+ parallel on my dev environment) and then trying to do things, in which case PHP is busy working on fetching the API results.

[OpenSource03]

I've tried to reproduce this and cannot, I really don't know what to tell you here.

I have my entire node offline, and everything continues working exactly as I expect it should, minus the /resources API calls that return the expected 504 errors from the API but that is simply a front-end issue.

I believe this might be an issue with your php-fpm config. I can reproduce similar results only when flooding the API endpoint with a ton of requests at once (100+ parallel on my dev environment) and then trying to do things, in which case PHP is busy working on fetching the API results.

Did you connect it to CloudFlare first?

I’m pretty sure it’s not due to Cloudflare config. Every time I ever installed this panel on any machine, I got the same behavior. The config is the default one. We’re currently running the panel as the only software on a whole 32GB 4 core 8 threads OVH rise baremetal.

[Wepwawetcode]

Can confirm that I have run into similar problems as @OpenSource03
I also use Cloudflare. When wings is unresponsive for one node. Then the panel becomes really slow.