Parsing .log files

[Adsa]

Hi..

 

I need some help with parsing a file.

I am parsing a log file to a db but the file is being written to all the time as well. I am wondering is there php command i can use so when i open the file i can go to the last parsed line of the .log file? or am i going to have to record the last line and match it the continue from there?

 

Thanks in adavance

[ldougherty]

If you use the file() function to read the file it will create an array, you could therefore count the elements in the array and start your next read at the line 1 greater than the size of the array itself.

 

I personally would suggest renaming the existing log file when parsing it so that any new data is written to a new file.  Therefore you are not worrying about reading for position over and over again.

[MadTechie]

$lines = file('myfile.log');
$lastline = $lines[count($lines)-1];

 

or

$lastline = readlastline('myfile.log');
function readlastline($file)
{
       $fp = @fopen($file, "r");
       $pos = -1;
       $t = " ";
       while ($t != "\n") {
             fseek($fp, $pos, SEEK_END);
             $t = fgetc($fp);
             $pos = $pos - 1;
       }
       $t = fgets($fp);
       fclose($fp);
       return $t;
} 

[Adsa]

Thanks for that i will give it a go.

 

Although since the file is being written to all the time i still need to find the last parse point of the file when i open it again.

[MadTechie]

The first option uses more memory where at the second one uses more CPU cycles, both depends on the file size, So remember to manage your files size by creating new log daily, weekly, monthly (whatever) or a fifo system

[Adsa]

<?php

$access_file_path = "file.log";
$last="18456";
$count =0
$handle = fopen("$access_file_path", "r") or die("can't open file");
while (!feof($handle)) {
$userinfo = fscanf($handle, "%s %s %s %s %s %s %s %s %s %s %s %s %s \n");
	if ($userinfo) {
		list ($field1, $field2, $field3, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $field11, $field12, $field13) = $userinfo;
		$output = "$field1 $field2 $field3 $field4 $field5 $field6 $field7 $field8 $field9 $field10 $field11 $field12 $field13";
		$output = rtrim($output);
		$count+=1;
		if ($count == $last) {
			$userinfo = fscanf($handle, "%s %s %s %s %s %s %s %s %s %s %s %s %s \n");
			if ($userinfo) {
				list ($field1, $field2, $field3, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $field11, $field12, $field13) = $userinfo;
				$output = "$field1 $field2 $field3 $field4 $field5 $field6 $field7 $field8 $field9 $field10 $field11 $field12 $field13";
				$output = rtrim($output);
				echo "$output"; // do something with output
				}
				$userinfo=NULL;
			}
		}
	$userinfo=NULL;
}
fclose($handle);
?>

[Adsa]

I changed my code slightly but still not able to get it to work.

 

<?php

//get end of file pointer
$fp = fopen("file.log","r");


//set file pointer to end
fseek($fp,0,SEEK_END);

// get pointer
$end_file = ftell($fp); 

// set start of file to scan from
$start_file = 18456;

// set pointer where to start scan
fseek($fp,"$start_file",SEEK_SET);

//display information
while (!feof($fp)) {
$userinfo = fscanf($fp, "%s %s %s %s %s %s %s %s %s %s %s %s %s \n");
	if ($userinfo) {
		list ($field1, $field2, $field3, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $field11, $field12, $field13) = $userinfo;
		$output = "$field1 $field2 $field3 $field4 $field5 $field6 $field7 $field8 $field9 $field10 $field11 $field12 $field13";
		$output = rtrim($output);
		}
	$userinfo=NULL;
}
fclose($fp);
?> 

[.josh]

use two files. Open the file that is continually being written to.  take remove line 0-x.  do what you will with the data, put it in a new file

when you go back to first file, the "last parsed point" will always be line 0.  take from pile 1, add to pile 2.