Hello, i have high LOAD problems...

[menntarra_34]

Hello, my server has high load problems...

I have  2 XEON E5506 cpus which means 2x4 cores = 8 , so when my load is more than 8, i should be worried... And now it is constantly at  about 16-24 !!!

I have 32gb memory. My my.cnf is tuned pretty much at the level where it perfect (all mysql performance scripts report that it is all OK)

What is strange is that my websites are working fine.

My memory swap is zero, also my cpu %wa is pretty low!

 

So i have no clue what could cause problems, please advise...

Also here is output from top:

top - 11:55:48 up 5 days, 25 min,  1 user,  load average: 16.63, 16.98, 17.10
Tasks: 450 total,  13 running, 436 sleeping,   0 stopped,   1 zombie
Cpu(s): 74.8%us, 18.3%sy,  0.0%ni,  1.5%id,  0.7%wa,  0.2%hi,  4.5%si,  0.0%st
Mem:  32950216k total, 32827592k used,   122624k free,   143960k buffers
Swap: 34996216k total,      296k used, 34995920k free, 22197412k cached

as you can see cpu usage is HIGH, what my question is: 18.3% sy is high too, isn't it? If yes, how can i found what is causing this high system usage?

 

Thanks in advance

 

[requinix]

And you think the problem is Apache? What does top show as the most intensive programs running?

[menntarra_34]

 

Well i have high mysql usage, but it does not makes this high load, just here is the example i recorded today:

 

 

See: mysql uses about 338% CPU, but that is OKAY! Load is 5.72:

 

top - 16:11:49 up 5 days,  4:41,  1 user,  load average: 5.72, 6.25, 7.04
Tasks: 448 total,   1 running, 447 sleeping,   0 stopped,   0 zombie
Cpu(s): 48.4%us, 15.2%sy,  0.0%ni, 20.8%id, 10.8%wa,  0.3%hi,  4.6%si,  0.0%st
Mem:  32950216k total, 32795368k used,   154848k free,   144844k buffers
Swap: 34996216k total,      192k used, 34996024k free, 24051068k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
5382 mysql     15   0 5683m 2.8g 4404 S 338.0  8.8 681:55.53 mysqld
14034 nobody    15   0  254m  31m  11m S 20.5  0.1   1:41.59 httpd
14160 nobody    16   0  253m  32m  13m D 14.9  0.1   1:37.69 httpd
14075 nobody    15   0  252m  31m  12m S 12.2  0.1   1:38.79 httpd
4798 nobody    16   0  658m 343m 288m D 10.9  1.1   1224:49 lighttpd
14142 nobody    15   0  260m  38m  11m S 10.3  0.1   1:41.42 httpd
14145 nobody    15   0  251m  31m  12m S  6.3  0.1   1:38.43 httpd

 

 

 

Here mysql uses: 283% , but load is 13.80 ( it just dropped from 17...)

 

top - 16:56:54 up 5 days,  5:26,  1 user,  load average: 13.80, 14.37, 11.52
Tasks: 448 total,  10 running, 438 sleeping,   0 stopped,   0 zombie
Cpu(s): 61.8%us, 13.5%sy,  0.0%ni, 14.2%id,  4.3%wa,  0.3%hi,  5.9%si,  0.0%st
Mem:  32950216k total, 32778408k used,   171808k free,   142180k buffers
Swap: 34996216k total,      192k used, 34996024k free, 23590200k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  RUSER    COMMAND
5382 mysql     15   0 5922m 3.0g 4408 S 283.9  9.6 795:20.24 mysql    mysqld
14126 nobody    17   0  254m  33m  13m R 51.0  0.1   2:05.92 nobody   httpd
14042 nobody    17   0  252m  32m  13m R 41.0  0.1   2:16.45 nobody   httpd
14185 nobody    15   0  255m  34m  12m S 21.2  0.1   2:17.98 nobody   httpd
13989 nobody    15   0  259m  38m  16m S 20.8  0.1   2:20.51 nobody   httpd
14138 nobody    15   0  255m  36m  13m S 18.2  0.1   2:23.18 nobody   httpd
14060 nobody    15   0  251m  31m  12m S 14.2  0.1   1:53.44 nobody   httpd
14107 nobody    15   0  253m  32m  13m S 11.2  0.1   2:06.62 nobody   httpd
4798 nobody    17   0  914m 457m 402m R  9.9  1.4   1237:19 nobody   lighttpd

 

 

So all in all my mysql uses 200-400 % cpu, but that is not a problem, have never been a problem... now these high loads are occuring and i don't know the cause...

 

Some additional info:

About apache: i have keepalive on, with keepalive timeout = 1

And here is my prefork info:

StartServers 256
MinSpareServers 256
MaxSpareServers 256
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 0 

[cags]

When our server load got silly high (and took the servers offline due to so much traffic basically ddos'ing the servers) it was down to Apache. We were under a lot of load in terms of hits and it was memory bound spawning Apache processes. This was down to the fact that an instance was being spawned for every resource on the page i.e. images, css, js (it was a Magento install so there were plenty). I would imagine you could drop the load significantly by either using something like Varnish to cache the assets, or by proxying the assets with lighttpd, or other lightweight server (both solutions essentially do the same thing and stop Apache processes being spawned so often).