[SOLVED] Proxy Requests/ Script Kiddy

[jaymc]

I am aware of an issue on my server where as I accidently left the apache proxy module enabled, after a few months my servers IP was indexed on a lot of proxy websites

 

A year on and Im still getting a load of hits, obviously not being processed by apache but they still count as a client connection

 

This is causing apache to max out on connections and spiral the server load out of control

 

I need a way to stop this, I cant change the IP address of the server, so thats out the question

 

Here is an abstract from apache server-status module so you can see what apache see's

 

Notice no IP address or request, I think the question marks and ..reading.. are spoofed headers

 

This may not even be the proxy issue? Perhaps my server is getting spammed to hell by a script kiddy

 

How can I stop this?

 

[attachment deleted by admin]

[tomfmason]

Are you sure that is invalid requests to the old proxy? Can you provide part of your access and error logs while the server is experiencing the high load? Next time you are having problems try the following to find out the real number of connections.

netstat -apn |grep :80 |awk {'print $5'} |wc -l

 

Also, have you tried increasing apache's MaxClients setting? You may also want to read this article from debian administration on some simple ways of optimizing apache.

[jaymc]

Hmmm

 

netstat -apn |grep :80 |awk {'print $5'} |wc -l

 

Showed 5711

 

Oh dear?

 

I cant up max clients, as it basically uses everything it can, if I set it to 700 it would hit that, but before it got to that, the server would be crippled anyway

 

Maybe it is not the old proxy issue, although I did see a get in there, with a valid IP address for http://python.org which is obviously nothing to do with my server, especially with it being a GET

 

Do you think my server is being blasted with spoof requests

 

Also, if I can get the PID of the apache request, how can i match that with a TCP connection, to get the IP!

[tomfmason]

Can you provide a sample from your access and error logs showing a few of these invalid requests?

[jaymc]

I attached an image in my first posts

 

use this link anyway

 

http://www.phpfreaks.com/forums/index.php?action=dlattach;topic=200537.0;attach=4475;image

 

How can there be 5711 tcp:80 connections

 

Server limit 200

maxClients 200

 

Should prevent that right?

 

 

By the way, nearly all of those 5000+tcp connections on :80 are in the state of TIME_WAIT

 

 

I dumped netstat -n to a text file, all 7000 lines, at a glance the IPs do not seem to be the same

[tomfmason]

Can you provide a sample from your access and error logs showing a few of these invalid requests?

 

just tail /path/to/your/error.log >> some_file.txt. Then you can remove any info you don't want us to see. I need to see examples of the log and your log format to be able to help

[jaymc]

I dont keep logs, slows the server down

 

However, server-status is practically LIVE LOGS?

 

If you want to see that, I can show you, although its perfectly clean apart from thouse ..reading.. connections in the screenshot

 

I have just found an IP address in my TCP logs, that appears over 1000 times on port :80

 

I just actually spoke to the guy whos IP it was, guarintee knows nothing about computers, is it possible he has some sort of spyware thats hammering my connection?

 

However, I banned his IP via .htaccess deny all, 0 connections in TCP which is great, however, still the ..reading.. connections flooding in

 

is ..reading.. a special apache thing or a blatant spoof

 

 

Edit Actually, I have found quite a few IP address which appear 30+ times in netstat log, is it 1 tcp connection for each request, e.g a page with 50 images on will require 50 seperate tcp connections?

[tomfmason]

I dont keep logs, slows the server down

 

You should be keeping logs!.. You don't have to log everything but you should, at the very least, keep an error log. if you rotate the logs there shouldn't be any performance issue. If the phpfreaks' servers with their high traffic can handle it, I am sure yours can too. Can you provide a few examples of invalid connections from netstat -apn |grep :80 ?

[jaymc]

How can I get invalid connections via

 

netstat -apn |grep :80

 

That just shows me 7000 connections, no relation to whats going on in apache

 

I have the PIDs of the apache spoofs, is there a way to trace that down to a tcp connection. I notice netstat -p does not display the PID

[tomfmason]

How can I get invalid connections via

netstat -apn |grep :80

You can't... I only asked you to do that to prove a point. See, this is exactly why you need error logs. You have no real way of viewing all of the invalid requests after the request is made. As it stands you can only view active request and can't really do much about it programmatically.  You should be able to easily scan the logs and find invalid requests. Then from there you can work on a plan of attack for preventing them in the future. let me know when you start using logs and I will try to help more

[jaymc]

I actually have error logs enabled, it was access logs I disable as it causes way to much i/o

 

I can see some weird stuff in the logs, here is example

 

[Thu Jun 05 16:18:17 2008] [error] [client 72.36.x.x] script '/var/www/cgi-bin/proxy.php' not found or unable to stat

 

[Thu Jun 05 16:25:11 2008] [error] [client 67.18.x.x] client denied by server configuration: /var/www/html/images/proxy5, referer: http://12e1b435e5/

 

 

Incedently, "/var/www/html/images/" is my first VHOST, which has a deny all on it, as in the past these requests where hitting my initial VHOST which was a forum, of which the home page had 24 queries running on it, hence it killed mysql

 

Its obvious its still going on, do you think this is the proxy situation still

 

I can give you the complete log file if you want over MSN

[tomfmason]

If the requests are only going to the first vhost you could give it a custom error log and use the following script(setup on a cron to automate it) to ban the offending ips after 5 requests

 

#!/bin/bash
# usage 
#    ./this_script search pattern log_file
#       all hosts with 5 or more matches for the given pattern will be banned
#    ./this_script all log_file
#       all hosts that appear more than 5 times in the given log file will be banned
# @author tomfmason

ban_file=/etc/hosts,deny

function ban_ip() {
  exists=`grep ${1} $ban_file`
  if [ ! "$exists" ]; then
     echo "ALL: ${1}" >> $ban_file
  fi
}

function parse_logs() {
  ret=`grep ${1} ${2} |awk '{print $8}'|tail -n+5|sort|uniq|tr -d [1]`
  for r in $ret; do
      ban_ip $r
  done
}

function ban_from_log() {
  ret=`awk '{print $8}' ${1}|tail -n+5|sort|uniq|tr -d [1]`
  for r in $ret; do
     ban_ip $r
  done
}

if [ "$1" = "search" ]; then
   parse_logs $2 $3 \;
fi

if [ "$1" = "all" ]; then
   ban_from_log $2 \;
fi

 

For example, say your new error log for that vhost is in /var/log/apache2/default_error.log you would use the script like this:

 

bash script_name all /var/log/apache2/default_error.log

 

That will search through the given log file and will ban all hosts that appear more than 5 times. Like I said that will only work if that log file only has invalid requests and you want to ban them from all access to the server

[jaymc]

Thats great

 

Looks like a solution to me

 

Thanks!