HTTPD using 99% CPU

[tommy.boucher]

Hi,

 

I'm usig APACHE 2.**, on Fedora Core, using PHP 5, and CGI Script (.pl).

 

In my virtualhost, I configure suexec for each web site, so when executing .pl file, I know who is using the CPU using the user, but when this is php (or maybe the problem is somewhere else), if the CPU for HTTPD is 99%, I can't know from where it come. Using Netstat I can see that my httpd openned a connection to a IRC server, but I can't find from each website of my server it's coming (I have around 30 web site) because HTTPD is running as "apache" user!

 

Can anyone help me? Solution?

 

Thanks a lot!

:'( :'( :'(

[trq]

What kind of virtualhost is this?

[JonnoTheDev]

Opens a connection to a IRC server!

Your server isn't being used for DDOS attacks is it?

[tommy.boucher]

Exampleof virtualhos:

<VirtualHost xx.xx.xx.xx:80>

        ServerName domain.com

        ServerAlias www.domain.com

        ServerAdmin support@domain.com

        DocumentRoot /data/SITES/www.domain.com/WWWROOT

        CustomLog "|/usr/local/sbin/cronolog /data/LOGS/www.domain.com/%y%m%d.log" IIS

        SuExecUserGroup domain.com domain.com

        <Directory "/data/SITES/www.domain.com/WWWROOT/">

                AllowOverride Options FileInfo AuthConfig Limit

                Options MultiViews SymLinksIfOwnerMatch IncludesNoExec

                php_admin_flag engine on

                php_admin_value open_basedir "/data/SITES/www.domain.com/"

                php_admin_flag display_errors on

        </Directory>

        ScriptAlias /cgi-bin/ "/data/SITES/www.domain.com/cgi-bin/"

        <Directory "/data/SITES/www.domain.com/cgi-bin/">

                AllowOverride None

                Options None

                Order allow,deny

                Allow from all

        </Directory>

</VirtualHost>

 

Probably used for DDoS or Spam shit, I really need to find from where it's coming!

[JonnoTheDev]

I think you may have to use a trial and error approach. When under a DDOS attack (massive server usage levels) I personally had to disable each site on the server one by one to determine where the attack was aimed at. You may have to do the same thing to find the location of the problem. I cant see a simple php script causing this issue, more like a 3rd party manipulating a script. I would check the access logs on each site to look at page request times and unusual patterns. Modification of the apache environment variables may have some undesired effects but if the server has been running fine before without any mods then you can rule this out.

[corbin]

It could be an IRC bot....  It's probably a terribly constructed while loop in their script.

 

I would just look for an IRC script and once you find it, ask the user to either remove it or recode it.

[JonnoTheDev]

Couldnt you set the max execution time in the php.ini to deter something like a badly constructed loop?

[corbin]

What about set_time_limit()?

[tommy.boucher]

Oki thx everyone for you comment,

 

the point is that there's around 200 web site on this server, I can't disable it one by one (when I disable one, I have to restart the service, and the attack stop... I can't wait until the attack's back.

 

Second point, I would like to install something to help me monitoring it, to know from where is it

 

It's probably a IRC bot, but how to find the script on the server, or how to find the php script that call the external script?

 

My backup script is using PHP so if I put "max execution time" to 1 min, it won't be enough for me server to backup =-)

 

Any idea?

[steviewdr]

Yep -

 

Put a firewall on the server and block outgoing traffic.

 

Typically a webserver *only* needs inbound over port 80 (and maybe 443).

If you block all external traffic (traffic going from your webserver outside to the internet), teh IRC script wherever it is will not be able to connect and thus fail, and hopefully should not use up your server resources.

 

If your running linux see: http://wiki.kartbuilding.net/index.php/Iptables_Firewall

The one line should do it:    iptables -A OUTPUT -j REJECT

 

If your on windows, you might have to get additional firewall software. I use(d) kerio which allows for fine grained control over ports.

 

-steve

 

[tommy.boucher]

This is what I did, but I really want to find from each website it's coming, where's the hole!!!!

 

Patching isn't always the better solution =-)

 

I found that the problem start after someone forgot the firewall down... but is there any way to monitor apache enough to know from where it's coming?

[corbin]

You could just search all files for mention of fsockopen, socket_create or 6667.

 

That should find the file, although it might take a few minutes to search through 200 sites' files.

[burn1337]

question, how many servers, or other computers/devices use the internet on your network? I have experiacned a few problems simular to that, although when I looked at my traffic I found no traffic, no outbound, no inbound. Shortly after I accendtly unplugged my server and my iptables stopped working, I still got the same thing but even still I find no outbound or inbound. Although I have found that yum was updating so seeing as your on fedora I would suggest checking that. Once I finally got yum to stop, then it stopped.