reading a log file one line at a time depending on timestamp

[eon201]

Hi,

 

Currently I am attepting to create a stats package for my companies website (yes my life would be 100% stress free if I was allowed to use a standard/commercial stats package ).

 

So far things are all good, and ive progressed quite far. But ive hit a snag.

 

At the moment im reading in the WHOLE log file and sorting it into an array and passing the neccesary data to mysql. Easy peasy.

 

But my boss has pointed out to me that as the site gets more and more traffic the log gets bigger, and if it gets to big the servers memory will overload, and we all know what happens then!

 

So what I need to do is read in the file one line at a time. and compare the timestamp to the last time the script was run. If the timestamp in the log is less than the last timestamp for when the file was run dont read it. If it is more than defiently read it.

 

so far my code looks like so...

 

$filename = '../../logs\W3SVC10019'. '/ex'. date('ymd'). '.log'; // Filename Checker
            
    if (file_exists($filename)) // Does the corresponding filename exist - if yes then
    {
        $fp = fopen($filename, "r"); //Open the server log
        $content = fread($fp, filesize($filename));     // Read the server log    
        $content = explode("\n", $content); // explode into array    
        $content  = array_reverse($content ); // reverse the array
        foreach($content as $key=>$value)    {

//This is where im sorting all the array data
}

So what needs to happen is that the script reads each line which look like...

 

2007-11-22 23:59:59 W3SVC10019 **.***.***.*** GET /images/e-br...

 

then grab '2007-11-22 23:59:59' and convert to timestamp ( strtotime )

then compare it to the mysql stored last run time of the script.

 

If the timestamp is less than the mysql version. Ignore

If it is more read it in.

 

How do I read one line at a time??

But only the first date/time part, and then figure out true/false??

 

 

I hope this all makes sense. Ive tried to explain this fully. If it all still does not make sense, please ask for more info!

 

Ok thanks in advance! Eon201

[trq]

Is this a Linux or Windows server?

[eon201]

Windows...

 

I thought that the fread would do it. But I cant yet find a way to do it.

 

And reversing a file read is perplexing me also. Reversing arrays no problem. But reversing the order in which you read the file? Mystery. ???

[trq]

Its just that there are heap of tools available on Linux that would make this job allot easier.

 

On windows however, your going to need to read the entire file into an array, there is no other way of doing it. Even just to do the comparisons you'll need to read the entire file.

[eon201]

Right. I appreciatte that. My boss is pretty adament that it has to be done this way. Is there any documentaion on this??? So i can PROVE it to him??

 

I do take your word for it. (otherwise im sure your status would not be super genius!)

 

But wiould the fgets function not work?

 

Or is there a way to read from the bottom up and stop when it hits a certain charcter match (eg the timestamp)??

[eon201]

Hi,

 

Im looking to read a file line by line backwards without loading the whole file to memory first.

 

So far I am this far (not very far!)...

 

if (file_exists($filename)) // Does the corresponding filename exist - if yes then
{
	$fp = fopen($filename, "r"); //Open the server log
        
	while (!feof($fp))
	{
	$Data = fgets($fp, 1024);
	// here is where i need to read the file backwards or reverse it!
	print $Data;
	$filetime = substr($Data, 0, 19);

	echo "<br/>$filetime<br/><br/>";
	}

 

So i can read the file line by line and find the info that I want with strpos. But I have to read it from the bottom up. (As the file is massive).

 

Ive tried array_reverse and strrev to no avail.

 

Can anybody please help me with this??? ???

 

 

[obsidian]

I don't believe there is any way to do that without loading it into memory. Using file() or something that loads it into an array and then calling array_reverse() would do it, though. The problem lies in that there is no way to tell what line to jump to without reading the whole file anyway. Also, the "file pointer" either has to start at the beginning or the end... You would be much better off to write your file backwards by prepending each line to the document rather than trying to read it backwards.

[eon201]

Unfortunatly I cant write the files backwards as im dealing with server logs.

 

And the most up to date data is at the bottom of the file.

 

Once i can read the file in backwards I can then loop through each line as I read it, matching timestamps, and as soon as one goes out of sync enough I can break the loop. Therefore saving HUGE amounts of memory.

 

Is there no way to store the file to an array before reading it, and then store it to an array???

 

 

Thanks for the help so far. Eon201

[eon201]

lol. Sorry didnt read your post properly.

 

How would I prepend the file?? Would this have to be loaded into the memory?? Or will it work on the fly?

[obsidian]

Think about your question above for a minute... to get the data of the file into an array, you have to read the file, otherwise PHP knows nothing about it. You can read it into an array without outputting it, but you still have a single array with the full contents of the file:

<?php
$lines = file($path);
$c = count($lines);
for ($i = $c - 1; $i >= 0; $i--)
{
  echo $lines[$i] . "<br />\n";
}
?>

[obsidian]

lol. Sorry didnt read your post properly.

 

How would I prepend the file?? Would this have to be loaded into the memory?? Or will it work on the fly?

 

No, your response was accurate, if you're loading server logs, you don't have control over prepending the log lines rather than appending them upon write time.

[obsidian]

One other thought... can you run exec() from your PHP code on your server? If so, why not just run an apache tail on the log file and retrieve the last few lines you want? If you use exec(), you can return the output as an array with the second parameter.

[eon201]

Ok. I appreciatte your time thus far.

 

So what options do I really have left?

 

I cannot load the whole file to memory (as the site is getting busier everyday if the server runs a script which loads the file avery 15 mins and the file is 200-300mb its curtains for me and the server!).

 

Yet I cant reverse it without doing so, leaving me reading the whole damn thing anyway just slower line by line!

 

This is actually the first time where I have come across a 'This cannot be done with php'. Woe is me.  :'(

[eon201]

Hmmm... Im not sure what you meant by the exec post.

 

Can you explain a little further for me please??

[eon201]

j i have just found this in the php manual...

 

I think that the quickest way of read a (long) file with the rows in  reverse order is
<?php
$myfile = 'myfile.txt';
$command = "tac $myfile > /tmp/myfilereversed.txt";
passthru($command);
$ic = 0;
$ic_max = 100;  // stops after this number of rows
$handle = fopen("/tmp/myfilereversed.txt", "r");
while (!feof($handle) && ++$ic<=$ic_max) {
  $buffer = fgets($handle, 4096);
  echo $buffer."
";
}
fclose($handle);
?>
It echos the rows while it is reading the file so it is good for long files like logs.
Borgonovo

 

Does it make sense to you Obsidian?

[obsidian]

Hmmm... Im not sure what you meant by the exec post.

 

Can you explain a little further for me please??

 

OK, first, check out your server and see if you can execute linux commands via the PHP exec() function. Some hosts turn off this functionality, but if it's on, this may be your best way to go about it. If you can use the function, passing in a variable for the second parameter will provide an array with the output of the shell command.

 

Next, you need to familiarize yourself with the LINUX tail command. Called with no parameters, it will, by default, return the last 10 lines of the file:

tail myfile.txt

 

However, you can pass in the -n parameter and tell how many lines you want. So, you could try something like this:

<?php
$command = "tail mylog.log -n 100";
exec($command, $res);
$lines = array_reverse($res);
foreach ($lines as $line)
{
  echo "$line<br />\n";
}
?>

 

This will output the last 100 lines of your file in reverse order... provided you change the "mylog.log" to your log file.

[obsidian]

Does it make sense to you Obsidian?

 

Fine to do, but unnecessary overhead if you're only needing the last several lines. Just how much of the files are you after? My last post will work up to a point, but if you're wanting thousands of records, you may want to do that sample.

[eon201]

Thats great ( I feel you have shown me the way now).

 

But I need to only read the files which have the correct date/timestamp compared to a saved one in the mysql.

 

eg... the log lines each start with.. 2007-11-26 00:00:25 (obviously a diff time each) ... id convert it to a timestamp, and then compare it to the mysql version for each line. when one doesnt match id break the loop.

 

Otherwise I dont know how deep (As shown in your suggestion) to read.