50 clients Reading for several minutes, or DOS attack?

[kevintynfron]

Hi all,

I’m having a very strange problem where the web server seems to hang for several minutes.

The actual server is just sitting there quietly, but the /server-status page shows all 50 of the 50 available Apache child processes as ‘R’.

 

Server Version: Apache/2.0.54 (Fedora)

Server Built: Aug 4 2006 14:38:17

________________________________________

Current Time: Wednesday, 19-Nov-2008 11:55:00 GMT

Restart Time: Sunday, 16-Nov-2008 03:40:58 GMT

Parent Server Generation: 1

Server uptime: 3 days 8 hours 14 minutes 1 second

50 requests currently being processed, 0 idle workers

RRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR

 

The single ‘W’ is, I assume, the child that’s writing the screen back to me.

 

The top command wasn’t showing anything as being busy (didn't notice if it was waiting I'm afraid), but the web server isn’t handing anything out. After a couple of minutes, the ‘R’s go away and normal service resumes.

 

What does the ‘R’ signify? Waiting for a client to finish asking for things? How do I/ Can I stop it happening.

 

Cheers,

Kevin

 

[corbin]

Could be KeepAlive being evil.

 

 

Try setting the MaxKeepAlive value lower or something.

[kevintynfron]

In what way evil?

 

at the moment,

 

KeepAliveTimeout 2

 

Thanks,

Kevin

[corbin]

The default keep alive value is 15 if I remember correctly, and the default max clients is 50 meaning for each connection, that connection (assuming it's a keepalive connection) is tied up for 15 seconds after the last request.  A waste of time for the connection.  That's how it's evil ;p.

 

 

Did the KeepAliveTimeout help?

 

 

"After a couple of minutes, the ‘R’s go away and normal service resumes."

 

 

I didn't see minutes the other day....  Is it literally minutes?

[kevintynfron]

Yes, minutes.

 

When nobody had the ability to have a look, they used to just reset the server, I'm guessing it was the same issue, and that resetting the server killed the 'R' processes and allowed normal ones to start again when it restarted.

 

At that point they had Maxprocesses set to 10 and keepalive off.

I now have 50 MaxProcesses and  keepalive turned on but at 2 seconds. It was doing what you explained with the default of 15s, so I turned it down). I would expect to see a load of 'K's if it was was that.

 

I haven't turned extended status on ,but looking at www.apache.org/server-status, an 'r' process doesn't even have a client IP address, so what's going on?

 

Thanks for the help,

Kevni

[corbin]

"R" Reading Request

 

 

You're right....  I don't think it has to do with keepalive then, since the IP would be known then.

 

Hrmmmmmmmm......Perhaps someone is trying to mess with your server.  It could be a badly written script somewhere or something.

[corbin]

OK, I came up with a plan, albeit not a very good one.  Since Apache won't tell you the IP, you can just look for the IP that's connected to your server on port 80 that's not listed on server-status.

 

 

Do:

 

netstat -na | find "80" | find "ESTABLISHED"

 

In cmd, and then the results will be like:

Protocol LocalAddress ForeignAddress State

 

So, you will want to look for all of the foreign addresses where the localaddress is 127.0.0.1:80 or 192.168.1.2:80 or something.  Then, you'll want to find the foreignaddress[es] that are not listed in the server-status.

 

 

I'm pretty sure it's either a crappy browser or a script gone wrong.  If I telnet my own server, and then don't do anything on the connection it sets there are R until the connection times out, which can be a super long time.  Wonder if there is a way to set Apache to kill connections after X seconds if nothing is written.

[kevintynfron]

That's great, thank you.

 

Unfortunately, I'm doing my proper job for most of the day, so don't get to see it in this state very often.

I shall attempt to fashion a script that others can run when the behaviour is noticed to a file that I can look at later.

 

Many thanks,

Kevin

 

[corbin]

Hrmmm you could just make it run every 10 minutes during the time that the problem usually occurs.  But, that might be more trouble than it's worth.

[kevintynfron]

This seems to an issue with the clients. Either they're behaving badly, or there's a DOS attack.  The number of R processes comes and goes during normal use, and I can see that when the server was set up to only have 10 processes max, it could easily saturate them with 10 'R' processes. The example I posted at the start of this thread had maxprocesses set to 50, and I've upped it to 75.

 

An 'R' process seems to be waiting in a FIN_WAIT_2 status, and there's notes on that  here.

 

The answer would appear to be to use the Timeout directive. The Apache Docs detail how it cuts off connections that are taking too long.

 

At the moment timeout is set to 120 seconds. I've switched server-status to use extended mode, so I can see a bit more detail. Watching the processes, every now and a gain, I'll get one that is stuck on 'R'. After 120 seconds, it gets killed off, which is what I want.

 

Monitoring the processes (although only  for about 10 minutes), 'normal' requests come in generally below 10 seconds (well, I don't see them as R processes), but there's a few that can take up to 30 seconds, so I think I'll try lowering it to 30 seconds for now.

 

Presumably the downside would be if there's a bandwidth issue, there could be a problem. Any ideas how files are uploaded to an apache server? Timeout will also affect "The amount of time between receipt of TCP packets on a POST or PUT request" So I may get problems if somebody's uploading stuff on a slow link.

 

Unfortunately, one of the tricky things about this is that I can't actually tell if I've solved it, just that it hasn't happened for some times, and as the problem was random to start with, I'll never really know.

 

Cheers,

Kevin