Random Connection Problems

[kavukattu]

Fellow PHP Freaks, I need some help!

Let me describe my setup first.

Dedicated Server, running Fedora Core 3, Plesk 7.5, Apache 2, PHP 4.3, MySql 3.x.

This server hosts a number of systems, all built using PHP. Some of the older ones aren't built so well, so are memory hogs, the more recent ones are built using Seagull and are nice and pleasant.

The core of our system is an XMLRPC Syncscript, that allows Palm devices to synchronize their data. So a single sync generates numerous tcp connections, and quickly eats up CPU and Memory.

In 99% of the cases, every thing is fine. People can sync with no problem, access our various websites without trouble. Every so often there will be lag while people are syncing, but it's always for a short period of time.

However now more often we are getting reports from users who are getting random connection problems. For instance, the XMLRPC server will at times return "nothing". But in all attemps, we cannot recreate the problem. From what we can determine, it is not a memory issue, and we do not feel it is a PHP timeout issue...as with all of our simulations, we cannot replicate. For instance, I whipped up a Java program that spawns X number of threads and simulates a sync with the server using httpunit. We can spawn up to 100 threads on a number of machines and have absolutely no problems!

And now recently, our websites have been getting reports of problems. Where people are reporting problems like "Cannot display page" and "cannot find server" while browsing the website. We monitor our servers with MRTG, and every so often we see TCP connection spikes, but memory and cpu are always quite sensible. And since our last reboot, we have not paged once.

We've scanned through logs, every so often we see apache automatically restarting...but it never coincides with an incident reported by a user.

Another thing that we have noticed, that may be related, is that we have noticed a lot of foreign connections to our SMTP server. Unfortunately this box also has to run mail services...and none of us are too excited to start configuring qmail.

If anybody has any ideas, it would be greatly appreciated!

Thanks,

Toby