Jump to content

memory_limit problems...

Carterhost

By Carterhost
in PHP Coding Help

 Share

Recommended Posts

Carterhost Newbie

Carterhost

Posted
Posted
I have the following code to parse my web server log file and display results for specific files.

I tried to run this the other day for a .gif that had over 5K hits, and ran into problems with memory_limit (i.e. the script had exceeded it)

How would I get around this. Would it be a case for [b]ob_flush()[/b]? if so, Where?

This is what I have so far:
[code]<?php
if (isset($_GET['file'])){$ss="/files/".$_GET['file'];}
$pageid= "Log Access";
// Full path to access log file on server:
$log="access_log";
// attempt to open the log file:
$lines=file($log) or die("Cannot open logfile");   
// page title:
$pageTitle="Transfer.carterhost.co.uk/logs";
// total lines:
$totalLines=count($lines);
// will hold lines according to status field of log line:
$all=$status_200=$status_404=$status_304=$status_other=array();
// will hold first and last times in log:
$firstTime=$lastTime="";
// START MAIN EXECUTION
###############
// Do the do:
parseLog_status($lines);
pageOpen();
doResults($all, $user);
pageClose();
// END MAIN EXECUTION
####################################
/////////////
// FUNCTIONS
/////////////
####################################
/******************************************************
* Function Name : pageOpen
*
* Task : build the page
*
* Arguments : none
*
* Returns : none
*
******************************************************/
function pageOpen() {
    global $ss,$pageid,$all,$status_200,$status_404,$status_304,$status_other,
            $totalLines,$firstTime,$lastTime,$pageTitle,$log;
?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>
      Access logs for
      <?=$ss?>
    </title>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
    <meta name="keywords" content="transfer,online,storage,free,webspace,webhosting,web,hosting,upload" />
    <meta name="description" content="Upload and Transfer files and photos, for free!" />
    <meta name="author" content="rob.j.carter" />
    <meta name="revisit-after" content="1 days" />
    <link rel="stylesheet" type="text/css" href="http://transfer.carterhost.co.uk/style.css" />
<script type="text/javascript" language="javascript">
var newWindow = null;
function closeWin(){
if (newWindow != null){
if(!newWindow.closed)
newWindow.close();
}
}
function popUpWin(url, type, strWidth, strHeight){

closeWin();

if (type == "fullScreen"){
strWidth = screen.availWidth - 10;
strHeight = screen.availHeight - 160;
}

var tools="";
if (type == "standard" || type == "fullScreen") tools = "resizable=no,toolbar=no,location=no,scrollbars=no,menubar=no,Width="+strWidth+",Height="+strHeight+",top=0,left=0";
if (type == "console") tools = "resizable,toolbar=no,location=no,scrollbars=no,width="+strWidth+",height="+strHeight+",left=0,top=0";
newWindow = window.open(url, 'newWin', tools);
newWindow.focus();
}
</script>
  </head>
  <body>
    <p>
      Reports for
      <span class="good">
        <b>
          <?php echo($_SERVER['HTTP_HOST'].$ss) ?>
      </span></b>
    </p>
    <!--<p>Access logfile: <?php //echo $log ?> </p>-->
    <p>
      Period:
      <?php print("<b>$firstTime</b> to <b>$lastTime</b>") ?>
    </p>
<?php   
    print("<p>");
    print("404 pages (page not found):        ".count($status_404)."<br>");
    print("200 pages (successful request):    ".count($status_200)."<br>");
    print("304 pages: (page not changed):     ".count($status_304)."<br>");
    print("other pages:                       ".count($status_other)."<br>");
    print("Lines in Log:                      ".count($all)."<br>");
    print("</p>");
    ?>
<?php
} // end func
/******************************************************
* Function Name : parseLog_status($lines)
*
* Task : parse each logfile line into array depending on status
*
* Arguments : $lines - lines from logfile
*
* Returns : none
*
******************************************************/
function parseLog_status($lines) {
    global $ss,$all,$status_200,$status_404,$status_304,$status_other,$firstTime,$lastTime,$totalLines;
    // inc counter - used to check if first or last item in logfile:
    $count=0;
    // parse each line:
    foreach($lines as $line){
        // trim spaces just in case:
        $line=trim($line);
        // split each line of log file at spaces - assumes log file fields are space delimited:
        $fields=split(" ", $line);
        // SERVER SPECIFIC:
        // LAST FIELD IN $FIELDS IS PID, POP THIS OFF STACK:
        array_pop($fields);
        // and assign each field a readable name:
        $host=$fields[0];
        $ident=$fields[1];
        $uname=$fields[2];
        $dateLong=substr($fields[4],1,strlen($fields[4]));
        // get time strings:
        $tmp=split(":", $dateLong);
        $date=array_shift($tmp);

        $UKtime=join(":", $tmp);
        $UKhr=($tmp[0])%24 ;
        $UKhr=str_pad($UKhr, 2, "0", STR_PAD_LEFT);
        $UKtime= $UKhr.":".$tmp[1].":".$tmp[2];
        $dateOffset=substr($datelong, 0, 5);
        $requestType=substr($fields[6], 1);
        $requestURI=$fields[7];
        $uaProtocol=substr($fields[8], 0, -1);
        $status=$fields[9];
        $bytes=$fields[10];
        $referer=substr($fields[11], 1, -1);   
        $ua=implode(" ", array_slice($fields, 12));
        // create array of fields for this line:
        $result=array(
                        $host,
                        $ident,
                        $uname,
                        $date,
                        $UKtime,
                        $requestType,
                        $requestURI,
                        $uaProtocol,                           
                        $bytes,
                        $referer,
                        $ua,
$status
        );
        // Push $result onto an array depending on $status:
            if (stristr($result[6], $ss) and !stristr($result[6], "log.php")){
            if (!stristr($result[6], "images")){
switch($status){
            case "200":
                array_push($status_200, $result);
                break;
            case "304":
                array_push($status_304, $result);
                break;
            case "404":
                array_push($status_404, $result);
                break;
            default:
                array_push($status_other, $result);
                break;
        }
//push results into $all array
array_push($all, $result);
}}
        // check if first or last line - if so get start/end time for period
        switch($count){
            case 0:
                // track first timestamp in log:
                $firstTime=$dateLong;
                break;
            case $totalLines-1:
                // track last timestamp in log:
                $lastTime=$dateLong;
        }
        // increment counter:
        $count++;
    }
} // end func
/******************************************************
* Function Name : doResults($title, $results)
*
* Task : display a table of results/log lines
*
* Arguments : string $title - title of result table
                array $results - array of lines to show
*
* Returns : none
*
******************************************************/
function doResults($results,$user) {
    global $ss,$filter;
   
    isset($filter)?"":$filter="All";
    if(!empty($results)){
        // print out results:
    ?>
    <br>
    <table class="style3" border="1" cellpadding="2" cellspacing="1">
    <tr>
    <th> Remote IP </th>
    <th> Usr </th>
    <th> Date/Time </th>
    <th> URI </th>
    <th> Protocol </th>
    <th> Bytes </th>
    <th> Referrer </th>
    <th> User Agent </th>
    <th> Status </th>

</tr>
    <?php
    foreach($results as $result) {
?>
<tr>
    <td><?php echo $result[1]; ?></td>
    <td><?php echo $result[2]; ?></td>
    <td><?php echo $result[3]."<br>(".$result[4].")"; ?></td>
    <td><?php echo wordwrap($result[6], 35, "<br />\n",true); ?></td>
    <td><?php echo $result[7]; ?></td>
    <td><?php echo $result[8]; ?></td>
    <td><a href="<?=$result[9]?>" target="_blank"><?php echo wordwrap($result[9], 35, "<br />\n",true); ?></a></td>
    <td><?php echo $result[10]; ?></td>
    <td><?php echo $result[11]; ?></td>

</tr>
<?php
        }
        print("</table>");
    } else {
        print("<p class='error'><b>There are no log entries for ".$ss."</b></p>");
    }
} // end func doResults($title, $results)
/******************************************************
* Function Name : pageClose()
*
* Task : close page - ending html tags
*
* Arguments : none
*
* Returns : none
*
******************************************************/
function pageClose() {
      ?>
      <p>
        <a href="javascript: self.close();">Click here to close this window</a>
      </p>
  </body>
</html>
<?php
}
?>[/code]
Link to comment
Share on other sites
btherl Member

btherl

Posted
Posted
You can increase your memory limit with:

[code=php:0]ini_set('memory_limit', '16M');[/code]


To increase it to 16 megabytes, for example.

I suspect the memory consumption is due to the array of results.  I don't see anything else that would consume memory.

An option to reduce memory consumption is to write your data to temporary files instead of storing it all in an array.  Or, you can write it out immediately rather than building an array, if that makes sense for how you are displaying the data.

For example, you could have a status_200 file, a status_304 file, status_404 file, status_other file and "all" file.  Then to do the final display stage you just read in each of those files and display the data immediately, consuming virtually no memory.
Link to comment
Share on other sites
This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.