Serious Apache crash issue

[sidd007]

Hey Guyz

Recently i have installed Uniserverz 11.0.2 in windows server 2012 R2. Apache crashes then and now and sometimes the Apache services does not start at all. Kindly find the logs below:

Windows Event log:

Faulting application name: httpd_z.exe, version: 2.4.23.0, time stamp: 0x577631b1

Faulting module name: ntdll.dll, version: 6.3.9600.18233, time stamp: 0x56bb4e1d

Exception code: 0xc0000409

Fault offset: 0x000257f9

Faulting process id: 0x199c

Faulting application start time: 0x01d2a9fbaf4a7ab2

Faulting application path: D:\UniServerZ\core\apache2\bin\httpd_z.exe

Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Report Id: 9d86b2d4-15ef-11e7-8105-005056b63d92

Faulting package full name:

Faulting package-relative application ID:

 

Apache error logs at the time of service restart:

[Fri Mar 31 09:54:10.620873 2017] [mpm_winnt:notice] [pid 6556:tid 504] AH00428: Parent: child process 6632 exited with status 255 -- Restarting.

[Fri Mar 31 09:54:10.699006 2017] [mpm_winnt:notice] [pid 6556:tid 504] AH00455: Apache/2.4.23 (Win32) configured -- resuming normal operations

[Fri Mar 31 09:54:10.699006 2017] [mpm_winnt:notice] [pid 6556:tid 504] AH00456: Apache Lounge VC14 Server built: Jul 1 2016 11:09:37

[Fri Mar 31 09:54:10.699006 2017] [core:notice] [pid 6556:tid 504] AH00094: Command line: 'D:\\UniServerZ\\core\\apache2\\bin\\httpd_z.exe -d D:/UniServerZ/core/apache2'

[Fri Mar 31 09:54:10.714633 2017] [mpm_winnt:notice] [pid 6556:tid 504] AH00418: Parent: Created child process 5528

[Fri Mar 31 09:54:11.480333 2017] [mpm_winnt:notice] [pid 5528:tid 424] AH00354: Child: Starting 150 worker threads.

[Fri Mar 31 09:54:13.355518 2017] [mpm_winnt:error] [pid 5528:tid 4500] AH00326: Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

Cannot initialize zend_mm storage [win32]

 

kindly reply back as soon as possible the issue has really become serious.

[Jacques1]

The error message seems pretty clear to me:

 

Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

 

It's also worth reading how Apache handles requests, so that you can adjust the configuration according to your requirements and hardware.

 

Last but not least: If this is a production website with a significant amount of requests, consider installing your own webserver (nginx, Apache, ...) from scratch. Those bundles are often made for newbies running quick tests rather than real applications.

[requinix]

I'm not sure the threads issue is the problem, though it running out of 150 threads within 2 seconds of starting is troubling.

 

The event log points to PID 6556 (the parent process) and 0xC0000409 is apparently a stack overflow. You're running a custom build of Apache so my first step would be to ask the Uniserver folks, which you already did.

Meanwhile the zend_mm error (from PHP) likely means you ran out of memory. And with the 150 threads... Are you stress-testing this?

 

If you're using Apache as an actual Windows service then try configuring the service to restart automatically so at least you won't have to do that.

[sidd007]

Thanks guyz for your advice...if I limit the number of child process will the application be able to handle that much load as I also have a doubt that too much creation of child process is consuming memory and creating orphan process so the processes are still running even when the application is down.

[Jacques1]

What kind of environment is this? I mean, are we talking about a public website with a significant amount of traffic? A small homepage? A local test setup? What's your hardware? A standard PC? A dedicated server?

 

And again, why use Uniform Server at all? The only reason I can think of would be convenience, but this is obviously irrelevant now. You've probably spent more time trying to debug this thing than any Apache installation would have taken.

[sidd007]

It's a production website which serves around 50000 request per minute. The website has to handle too much load. We are using Windows server 2012 r2 64 bit with 16gb RAM.

[requinix]

One server is not enough.

[Jacques1]

You definitely need a more professional software setup from a sysadmin who actually knows what he/she is doing. Running a production site on some MAMP-for-dummies test server is a bad idea.

[sidd007]

OK....let me see what we can do.

[sidd007]

hi guys.... i changed MaxRequestPerChild from 0 to 10000. Though i dont see the following error in error logs: Cannot initialize zend_mm storage [win32]. But now my error log has following entry:

[Tue Apr 04 18:38:50.756781 2017] [mpm_winnt:notice] [pid 5980:tid 500] AH00424: Parent: Received restart signal -- Restarting the server.
[Tue Apr 04 18:38:50.850542 2017] [mpm_winnt:notice] [pid 5980:tid 500] AH00455: Apache/2.4.23 (Win32) configured -- resuming normal operations
[Tue Apr 04 18:38:50.850542 2017] [mpm_winnt:notice] [pid 5980:tid 500] AH00456: Apache Lounge VC14 Server built: Jul  1 2016 11:09:37
[Tue Apr 04 18:38:50.850542 2017] [core:notice] [pid 5980:tid 500] AH00094: Command line: 'D:\\UniServerZ\\core\\apache2\\bin\\httpd_z.exe -d D:/UniServerZ/core/apache2'
[Tue Apr 04 18:38:50.850542 2017] [mpm_winnt:notice] [pid 5980:tid 500] AH00418: Parent: Created child process 684
[Tue Apr 04 18:38:51.772500 2017] [mpm_winnt:notice] [pid 684:tid 424] AH00354: Child: Starting 150 worker threads.
[Tue Apr 04 18:38:53.944570 2017] [mpm_winnt:error] [pid 684:tid 4068] AH00326: Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
[Tue Apr 04 18:39:43.105009 2017] [mpm_winnt:notice] [pid 5980:tid 500] AH00422: Parent: Received shutdown signal -- Shutting down the server.
[Tue Apr 04 18:39:44.589534 2017] [mpm_winnt:notice] [pid 684:tid 424] AH00363: Child: Terminating 134 threads that failed to exit.
[Tue Apr 04 18:39:44.589534 2017] [mpm_winnt:notice] [pid 684:tid 424] AH00364: Child: All worker threads have exited.
[Tue Apr 04 18:39:44.980166 2017] [mpm_winnt:notice] [pid 5980:tid 500] AH00430: Parent: Child process 684 exited successfully.
[Tue Apr 04 18:39:46.464680 2017] [mpm_winnt:notice] [pid 6136:tid 500] AH00455: Apache/2.4.23 (Win32) configured -- resuming normal operations
[Tue Apr 04 18:39:46.464680 2017] [mpm_winnt:notice] [pid 6136:tid 500] AH00456: Apache Lounge VC14 Server built: Jul  1 2016 11:09:37
[Tue Apr 04 18:39:46.464680 2017] [core:notice] [pid 6136:tid 500] AH00094: Command line: 'D:\\UniServerZ\\core\\apache2\\bin\\httpd_z.exe -d D:/UniServerZ/core/apache2'
[Tue Apr 04 18:39:46.464680 2017] [mpm_winnt:notice] [pid 6136:tid 500] AH00418: Parent: Created child process 3428
[Tue Apr 04 18:39:47.277239 2017] [mpm_winnt:notice] [pid 3428:tid 424] AH00354: Child: Starting 150 worker threads.
[Tue Apr 04 18:42:52.588079 2017] [mpm_winnt:notice] [pid 3428:tid 424] AH00358: Child: Process exiting because it reached MaxConnectionsPerChild. Signaling the parent to restart a new child process.

 

Apache services are still restarting.Can you guyz just guide me once more on this.

Edited April 4, 2017 by sidd007

[Jacques1]

I don't think you understand what we're saying.

[sidd007]

ok... i will. But what does this error mean:  Parent: Received shutdown signal -- Shutting down the server.

[requinix]

Check your access logs for anything unusual around that time. If it's not that then it seems like the OS is shutting it down, which is either a user-initiated action (can't help you there) or an automatic action (suggesting the process is doing Something Bad).

[sidd007]

Hey guyz

I am in the process of migrating to native apache installation. But while i was going through the error logs i found this error:

PHP Fatal error:  Out of memory (allocated 786432) (tried to allocate 65536 bytes) in D:\\UniServerZ\\www\\aav.php on line 216,

I guess this happens because some PHP script is eating too much memory, but still i  want your expert thoughts on this.

BTW, as i mentioned earlier my application is supposed to handle huge load, so what would be the appropriate 'ThreadsperChild' and 'MaxRequestPerChild' directive setting? Setting 'MaxRequestPerChild' to 0 wont work, will it?

[requinix]

Frankly, the fact that you have this server handling up to 1000 requests per second is astounding. I can only assume the situation is worse than you think it is.

 

PHP Fatal error:  Out of memory (allocated 786432) (tried to allocate 65536 bytes) in D:\\UniServerZ\\www\\aav.php on line 216,

I guess this happens because some PHP script is eating too much memory, but still i  want your expert thoughts on this.

Your server is running out of memory. You need more servers. You should have more simply for fault tolerance.

 

BTW, as i mentioned earlier my application is supposed to handle huge load, so what would be the appropriate 'ThreadsperChild' and 'MaxRequestPerChild' directive setting? Setting 'MaxRequestPerChild' to 0 wont work, will it?

Apache's MPM works by having some number of things handling requests. On Windows (mpm_winnt) there is one child and the "things" are threads.

 

ThreadsPerChild will restrict how many concurrent requests the server can handle. MaxConnectionsPerChild (modern name for MaxRequestsPerChild) is used to limit the impact of memory leaks, which are reclaimed when the process dies, but with mpm_winnt means the server will not be processing requests until the new child is ready. PHP isn't perfect and sometimes there are leaks, so if you use this setting then make it high by profiling memory usage and determining a) whether there are apparent leaks, b) how much those leaks will impact memory usage on your server, and c) how many requests the server can handle before memory is exhausted.

[sidd007]

What does this error means: AH00530: initialisation: bug or getaddrinfo fail?

[requinix]

Seems like it's a configuration issue of having the wrong IP address somewhere.