madz Posted March 9, 2007 Share Posted March 9, 2007 Hi all, I'm quite new in PHP. I want to make a log parser using php that can insert the parsed data into database (in this case I'm using MySQL). The log file is keep increasing (dynamically changing). As I can't find how to detect a change in a file, I just let the parser to be refreshed every few second, re-read all the log entry, parse it again and try to insert into database. I set the database so that no duplicate entry can be inserted. The parser works and every entry inserted correctly into database. However, I found that this kind of parser is not efficient as it parse the whole file again and do a database insertion everytime it refreshed. Is it possible to detect a change in a file so that the parser just need to parse the new entry instead of the whole file? Can anyone help me? Thanks before. regards, Madz ps: sorry for my bad English, I hope you can understand Quote Link to comment Share on other sites More sharing options...
skali Posted March 9, 2007 Share Posted March 9, 2007 May be after you have parsed the file once you can add a pointer in the file yourself at the last line your parser read. So the next time you will read the file you will only read from the last pointer that you had set and will not need to parse the whole file again and again. Just check for the last pointer set each time. Quote Link to comment Share on other sites More sharing options...
madz Posted March 12, 2007 Author Share Posted March 12, 2007 Hi, thanks for the response. Is it possible to make a pointer to a file in PHP? Could you give some code example? Anyway, if the pointer to the file is a variable, then it will be deleted from memory once I close the .php file isn't it? So what should I do to keep the pointer there, so that when later I want to parse the file again, I still have the mark to the last parsed line of the file? Thanks before. Quote Link to comment Share on other sites More sharing options...
btherl Posted March 12, 2007 Share Posted March 12, 2007 I would solve this by storing the last processed line number somewhere. You can store it in a file, or in the database. Then when you re-read the file, you skip lines until you get past the last processed line number. Alternatively you could store a byte offset from the start of the file, which allows you to fseek() to the exact location that any new lines will start. http://sg2.php.net/manual/en/function.fseek.php Quote Link to comment Share on other sites More sharing options...
madz Posted March 14, 2007 Author Share Posted March 14, 2007 Thanks to skali and btherl Since I'm using the array to store the file input (using file() method) I think I can't use fseek to remember the pointer to the last line. I decide to use a txt file to store the array index of the last line parsed. The text file will be read before parsing to retrieve the pointer, and then after parsing, the new pointer is written into that file. Below is the code : <?php // $i is index of the array $recFile = "recFile.txt"; if(file_exists($recFile)) { $tmp = fopen($recFile,"r") or die ("failed to read!"); $i = fread($tmp,filesize($recFile)); fclose($tmp); } else $i=1; while($i<$sizerecs) { //parsing here... } $tmp = fopen($recFile, "w"); if($tmp) fwrite($tmp, $i); else echo "failed to write!"; echo filesize($recFile); fclose($tmp); ?> Well, that's all my code. Maybe not so effiecient, as I'm just a novice programmer >.< If anyone wants to comment you're welcome =) Thanks. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.