[SOLVED] Parser for Dynamically changing file

[madz]

Hi all,

 

I'm quite new in PHP. I want to make a log parser using php that can insert the parsed data into database (in this case I'm using MySQL). The log file is keep increasing (dynamically changing). As I can't find how to detect a change in a file, I just let the parser to be refreshed every few second, re-read all the log entry, parse it again and try to insert into database. I set the database so that no duplicate entry can be inserted. The parser works and every entry inserted correctly into database.

However, I found that this kind of parser is not efficient as it parse the whole file again and do a database insertion everytime it refreshed. Is it possible to detect a change in a file so that the parser just need to parse the new entry instead of the whole file? Can anyone help me?

Thanks before.

 

 

regards,

 

Madz

 

ps: sorry for my bad English, I hope you can understand

[skali]

May be after you have parsed the file once you can add a pointer in the file yourself at the last line your parser read. So the next time you will read the file you will only read from the last pointer that you had set and will not need to parse the whole file again and again. Just check for the last pointer set each time.

[madz]

Hi, thanks for the response. Is it possible to make a pointer to a file in PHP? Could you give some code example?

Anyway, if the pointer to the file is a variable, then it will be deleted from memory once I close the .php file isn't it? So what should I do to keep the pointer there, so that when later I want to parse the file again, I still have the mark to the last parsed line of the file?

Thanks before.

[btherl]

I would solve this by storing the last processed line number somewhere.  You can store it in a file, or in the database.  Then when you re-read the file, you skip lines until you get past the last processed line number.

 

Alternatively you could store a byte offset from the start of the file, which allows you to fseek() to the exact location that any new lines will start.

 

http://sg2.php.net/manual/en/function.fseek.php

[madz]

Thanks to skali and btherl 

 

Since I'm using the array to store the file input (using file() method) I think I can't use fseek to remember the pointer to the last line. I decide to use a txt file to store the array index of the last line parsed. The text file will be read before parsing to retrieve the pointer, and then after parsing, the new pointer is written into that file. Below is the code :

 

<?php
// $i is index of the array
$recFile = "recFile.txt";
if(file_exists($recFile))   {
    $tmp = fopen($recFile,"r") or die ("failed to read!");
    $i = fread($tmp,filesize($recFile));
    fclose($tmp);
}
else $i=1;

while($i<$sizerecs) {

    //parsing here...

}
$tmp = fopen($recFile, "w");
if($tmp) fwrite($tmp, $i);
else echo "failed to write!";
echo filesize($recFile);
fclose($tmp);
?>

 

Well, that's all my code. Maybe not so effiecient, as I'm just a novice programmer >.<

If anyone wants to comment you're welcome =)

 

Thanks.