Apache slowly consumes all system memory while running streaming CGI script.

[JohnK]

Hello, I'm trying to solve a problem where some of our long-

running streaming CGI scripts are causing Apache to grow continuously,

eventually consuming all system memory and starving Apache and other processes

(manifested as Apache returning HTTP 500 (internal server error) responses to

subsequent requests to other resources, and other weirdness). I've reduced the

test to a really simple CGI script which reproduces the memory growth reliably

for me (see steps to reproduce below).

 

I'm wondering if this is normal behaviour (is continuous streaming not

supported?), or perhaps I'm just mangling my headers or missing something

really obvious. Any insight you have would be great.

 

 

I've observed the behaviour on both of our setups:

 

Linux: 2.6.13.4.D400-LEAF (a custom (i.e. suspect) roll-our-own Linux - gcc

version 4.0.2 20051125, Red Hat 4.0.2-.

Apache: 2.0.55

 

Linux: Ubuntu Fiesty (standard, nothing special)

Apache: 2.2.3

 

Steps to reproduce:

-------------------

1. Restart Apache.

2. Point a browser at the CGI script (see offending cgi script (1) below)

running either runPlain() or runMultipart().

3. Run ps every 10 seconds for 10 minutes (see ps log script (2) below).

 

My observations:

-----------------

1. In both the plain/text and multipart case, I observed Apache's memory

growing linearly at a rate of ~19.6kB/min (typically in 4kB increments

depending on your Linux/MMU page size).

 

2. When I stopped the script after 5 minutes, Apache did not appear to free its

memory. Then, when I restarted the script, Apache ran flat for 5 minutes before

starting to grab more memory. This suggests that when a CGI script stops,

Apache frees some of its internal memory pool, but does not release it to

system memory, instead reusing it for future CGI scripts. I found I had to

restart Apache after each test to avoid masking the memory growth of subsequent

tests.

 

3. I tried various data rates in my below scripts, by adjusting the sleep time,

chars per line and lines per cycle, and found the memory growth rate was

roughly a linear function of the CGI's data output rate (with the function

flattening at high data rates probably due to CPU/network saturation).

 

4. I tried various other stabs in the dark, like flushing stdout on each cycle

(before the sleep). No change to the growth rate.

 

1. Offending cgi script:

------------------------

#include <time.h>

#include <dirent.h>

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <sys/time.h>

#include <sys/stat.h>

#include <sys/types.h>

#include <unistd.h>

 

/*

main.

*/

int main(int argc, char *argv[])

{

    runPlain();

}

 

/*

Run a text/plain test.

*/

int runPlain()

{

    int i,j;

    printf("Content-type: text/plain; charset=us-ascii\n\n");

    for (i=0; ; i++)

    {

        for (j=0; j<5; j++)

        {

            printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka\n",i,j);

        }

        printf("\n");

        usleep(5000);

    }

}

 

/*

Run a multipart/mixed-replace test

per http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html section 7.2.1.

*/

int runMultipart()

{

    int i,j,num;

    printf("Content-type: multipart/mixed-replace; boundary=myboundary\n\n");

    printf("Preamble\n");

    for (i=0; ; i++)

    {

        printf("--myboundary\n");

        printf("Content-type: text/plain; charset=us-ascii\n\n");

        for (j=0; j<5; j++)

        {

            printf("%d,%d: wakka wakka wakka wakka wakka wakka wakka\n",i,j);

        }

        printf("\n");

        usleep(5000);

    }

}

 

2. ps log:

----------

#!/bin/sh

 

interval=$1

while :

do

  timestamp=`date +%Y.%m.%d-%H.%M.%S`

  ps -o pid,pcpu,mem,cmd | sed "s/^/$timestamp  /"

  sleep $interval

done

 

[PFMaBiSmAd]

Look into if your Apache is compressing output gz/gzip or has mod_gzip present.

 

It sounds like it is buffering the output until it has it all so that it can create the compressed content.

[JohnK]

Thank you for your suggestion however the web browser is receiving continuous data.  I agree with you that apache is buffering the data or at least part of the data stream.

 

[gattusunil]

Hi,

I am also facing the same problem while streaming files .. has nay body got solution for this.. Its urgent for me.....