Need some help with best practice for upload/parse/dump to MySQL of large log fi

[sephtin]

Looking for some best practice type of advice... and whether or not what I'm trying to do is feasible.  Maybe I'm approaching this incorrectly, and someone could point me in the right direction!

 

Here's what I'm trying to do.

I regularly receive log files, 2-3 on average a week.  The files are around 10+mb, and average around 200,000 lines.  My goal is to parse the log file, and dump appropriate parts of it into a load table in MySQL so that web pages can create reports from these log files.

 

My current process is to run a perl script on the log file that reads the file line by line in a while loop.  Each line is parsed for some regex matching and substitution (approx. 25 changes).  The resultant line is then dumped into a MySQL load table.  At that point, I'm done with the data, as it is manipulated from within the DB, and reported on from there.

 

Currently this is done localy on a Win32 workstation connecting remotely to MySQL, but I'd like to automate this a bit.  What I'd like to do, is have a page where the files could be uploaded, parsed, and automagically entered into the DB.  This would happen using PHP/Apache on Linux.

 

On the linux environment with (much smaller) test files, I've been able to create a form to upload the file, a page that parses the file, and currently I'm just spitting it onto a web page.  Entering into the DB should be easy, I just haven't gotten that far. 

 

My question is regarding best practice.  In Perl, it is suggested that the file be parsed line by line due to it's size (rather than loading the entire file into an array, etc.).  What is the most efficient method to do this in PHP?

The method I've been testing with is using file_get_contents() to set the file to a string, and parse it that way.  Wondering if this is the best approach, or if there might be a better way.

 

Also, if this is something that PHP can't do very efficiently (very possible).. another option might be to call a modified version of my perl script (?).. could use a pointer on how this can be done from php...

 

Please let me know if there are details I've glossed over. 

I appreciate any help/feedback!

Thanks,

-John

sephtin --AT--  gmail *&^d0t))(  com

 

[sephtin]

Wow, I'm sure this has been done before...

 

I would have thought this was something that could easily be done in php... ?

 

[Orio]

Try reading this tutorial.

 

Orio.