Apache 2.2.4/2.2.9, Windows 2003, Event 2019, High File Handles

[papkh]

Hi,

 

I have Apache 2.2.4 installed on three different Dell PC's, and I have Apache 2.2.9 installed on a fourth Dell PC. All are running Windows Server 2003.

 

They all cause an Event ID 2019 Error in Windows periodically. Meaning Windows has run out of memory in the nonpaged memory pool. (How often is related to how much traffic, but usually about every three days).

 

I've worked with Microsoft's tech support, running performance monitor logs and pool monitor logs.

Httpd.exe seems to have a memory leak, and is using a very high number of file handles that are never returned to the pool. (350,000 + before Windows basically hangs with the 2019 error).

 

Microsoft says look to Apache for answers. I've googled and googled.

 

The servers also run Php, but I've removed the Php from the httpd.conf file and watched the memory pools, and I think its a problem with or without Php.

 

Early on I suspected the rotatelogs.exe, but we removed that. Back then the server hit the 2019 about once a day, but after removing rotatelogs, it was once every three days. So rotatelogs.exe made the problem occur sooner, but doesn't seem to be the cause.

 

Does anyone have any experience with something like this?

 

Thanks.

-PKH

[corbin]

I don't know what it is, but I seriousllllyyyyyyy doubt there's a memory leak that big (not saying there is for sure one, just saying probably not one that big) in Apache.

 

 

 

What is your site setup?  What kind of site is it?  What kind of files does it access the most?  The more information you can give us the better.

[papkh]

Hi Corbin,

 

Thanks for the response.

The sites are pretty standard Apache / Php retail shopping cart sites. All in house code.

We use db2 on an as400 for the database. So that's probably a step outside the norm.

 

Here's what might be another interesting piece of info.

Whenever the server has crashed with a 2019, we see the file handles for httpd.exe high - like at 350,000, but we also see the used non-paged memory for that Apache process at 4 gig (all of the server's ram?).

 

As far as I know, the non-paged memory used by that Apache process has always remained low - around 50k give or take 10 or 20k depending on which server. Up until it crashes, where it seems to leap suddenly to the 4 gig mark.

 

We're running a graceful restart daily to keep the servers from crashing. And we thought that was working, because the file handles are reset with each restart. But we just noticed, 45 minutes after our last restart, even though the file hanldes are low, say 10,000, the non-paged memory pool is at 4 gig. We haven't seen a problem with the server yet and aren't getting the 2019 yet. But we're watching and anticipating that it will, even without the high file handle count crash? But we'll have to wait and see for sure.

 

Let me know if you see anything in all of that.

 

Thanks,

PKH

 

-PKH

[corbin]

The 4GB memory could be due to a strange 'feature' that Apache has where when it gets a hold of RAM, it latches onto it.  Then again, that should stop wayyyyy under 4GB.

 

 

 

I'm trying to think of why Apache wouldn't be closing file handles and I can't think of anything.  Maybe requests aren't terminating, or a PHP script is holding files open.  Hrmmm....  No idea x.x.

[steviewdr]

You will have to tweak httpd.conf.

 

Apache will use up all available memory unless you tell it not to.

 

Something like the following in httpd.conf should suffice:

<IfModule mpm_prefork_module>

    StartServers          5

    MinSpareServers      5

    MaxSpareServers      10

    MaxClients          150

    MaxRequestsPerChild 1000

</IfModule>

 

 

-steve

[corbin]

I thought it automatically would kill off children.  Guess the default threads (or processes) is higher than I thought.  (Or that guy is getting a lot of traffic.)

[papkh]

Hey steviewdr,

 

Thanks for the post.

I was excited with the concept, but in reseaching I think I'm seeing that Windows only allows the winnt mpm... that you can't use the preform mpm with Windows.

 

Wanted to point that out and see what you think there. Did you catch that I'm on a Windows server? Am I looking at older documentation on the web when I'm seeing that you can't use the prefork mpm with Windows? If you can load the prefork mpm in Apache for Windows, do you know how?

 

Anything else you can offer on this is greatly appreciated!

 

Thanks,

papkh

[papkh]

x

[corbin]

http://httpd.apache.org/docs/2.2/mod/mpm_winnt.html

 

 

Under Windows, mpm_winnt is your only option.