0

I have a Centos 7 server on which I installed apache 2.4.58, it has been crashing for 2 days after a short time that it is online, I keep stopping the apache service (systemctl stop httpd.service) and then restarting it but it remains online for an indefinite time and then starts to be unreachable causing 504 errors. I analyzed the access log but apart from the crawlers and users I do not see anything strange, to analyze the logs I am using goaccess.

Could this be caused by a DDoS attack? Or is it simply too many requests to handle?

Furthermore, the following apache modules are enabled: mpm_event mod_security mod_deflate mod_evasive

None of these are able to help manage these strange "spikes" of unknown traffic.

I also checked the server resources using the top and htop commands but even those do not report anything relevant.

Considering that the server has 16GB of RAM, I set these parameters in mpm_event:

StartServers 250
MinSpareThreads 250
MaxSpareThreads 500
ServerLimit 1000
MaxRequestWorkers 1000

What can I do to understand where and when this sudden spike in requests that crashes the system happens?

php-fpm error log:

[01-Jun-2024 14:09:39] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 4 idle, and 41 total children
[01-Jun-2024 14:09:52] WARNING: [pool www] server reached pm.max_children setting (50), consider raising it
[01-Jun-2024 15:25:08] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 4 idle, and 47 total children
[01-Jun-2024 15:25:09] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 2 idle, and 48 total children
[01-Jun-2024 15:25:10] WARNING: [pool www] server reached pm.max_children setting (50), consider raising it
[01-Jun-2024 15:45:28] NOTICE: Terminating ...

Another thing I noticed is that in the access log it is full of "bots" with user-agent: facebookexternalhits, so what I did was add a rule in mod_security to limit access to this bot.

I attach part of what I analyzed with goaccess after about 20 minutes from the restart of httpd: https://i.sstatic.net/IxbbPPvW.png

Improve this question
4
  • Please edit your question to add details. What symptom led you to believe that httpd was crashing? Did the main PID die and the systemd service appear dead, did the web site have user visible problems, was there evidence in the log file? Speaking of log files, what did you find, in both error and access? Can the problem be reproduced with httpd serving a static file, or is it all php?
    – John Mahowald
    Commented Jun 2 at 0:48
  • The symptom that httpd is crashing is that every vhost I access automatically becomes a 504 gateway timeout. This has been happening for 2 days and I can't figure out why, now I attach more on the access_log.
    – Matteo Cigognini
    Commented Jun 2 at 6:51
  • t seems that Apache is not the problem as replies are still being sent (504). The limits of the php-fpm subsystem are reached and no more processing takes place. Implement appropriate firewall/application proxy rules.
    – M. Behrens
    Commented Jun 2 at 18:51
  • Not a single Apache error log, only php-fpm logs showing limits have been reached. Apache is clearly reporting that backend is failing but your conclusion is that apache is crashing... Something is really wrong with the diagnostic. Also mod_security and mod_evasive are third party modules, If I were you I would not still use them until I get to know more what I´'m doing.
    – Daniel Ferradal
    Commented Jun 3 at 19:35

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .