Does such a product exist??

[nick1]

Fellow friends in PHP,

 

I turn to you in great frustration regarding my search for a general-purpose log file analyzer.

You're probably saying "why are you posting about log analysis here, idiot."

Well, because since it appears that I cannot find a log analysis product that fits my needs, I'm tempted to write a PHP script that does fit my log analysis needs and I want your feed back on if there is a better way to go about this madness.

 

Here's what my ideal log analyzer would do:

 

-Apache: report all information about who is accessing my web server (IP, time, browser, OS, etc).  Also report on any errors.  For this information I can look to AWstats, Webalizer, etc. But there's a lot more information I want reported.  Keep reading.

 

-MySQL: report who is accessing the database, time of access, queries used, any errors, etc.

 

-PHP: report any errors that might have occurred.

 

-SSH: report who is accessing the SSH server, time of access, IP the server was accessed from, etc.

 

-IPtables:  report on successful and denied requests, scans, attacks, etc.

 

-System logs: report on any operating system errors, etc.

 

It would be ideal if such a program could generate one master report, each section of the report containing information about a different log file. I'm developing an administrator webpage for my website where I can edit my database and *hopefully* setup my theoretical log analyzer to generate a master report on the criteria above, at the click of a button.  That's where the PHP script comes into play - it IS the log analyzer

 

Please tell me such a product exists.  If not, what magnitude of a project are we talking here?

Again, if there's a better way to go about all of this, please, do tell.  All feedback is welcomed.

 

Thanks,

 

Nick

[trq]

I don't think such an all in one product exists, and to be honest, I wouldn't think PHP would be the best tool for the job.

 

The whole thing sounds pretty customized really but all the applications you've listed already have logging ability built in, so really, its just a matter of parsing said log files.

 

Id'e be more inclined to use a combination of bash, awk, sed and grep or even Python or Perl for such a task. While php does have file manipulation abilities, those languages are much stronger in this area. Once your report has been made, if you wish to display it in a webpage via php move it to a format like XML or even a database and let php read it from there.

[Snooble]

thorpe = genius

 

I suggest Perl... as there is a demand for perl at this moment in time. C++ is another alternative.

 

Snooble

[nick1]

thorpe and Snooble,

 

thank you for your thoughts and suggestions, I appreciate them.

I too have considered using a more "robust?" language to accomplish this task.

PHP is probably my most comfortable language at the moment, but this project could be a good excuse for me to learn another language, such as Perl or perhaps C?  I was wondering if you can recommend any specific books on programming that are really excellent learning tools and not just another programming book?  Walking into the programming section of a book store can be like walking into the amazon without a guide... except the books cost more.  Thanks for your input,

 

*Nick*

[trq]

What you really need to look at is the task at hand. While PHP can read and manipulate files there are other languages around that are much stronger in this area. C is definitely NOT one of them.

 

Perl is well known as a text file manipulation language, the name itself (Practical Extraction and Reporting Language) suggest so. However, newer languages such as Python and even Ruby intentionally borrow the best bits from Bash, Sed, Awk, Grep and Perl and are IMO just as (if not more) powerful in this area (ps: I'm no Perl programmer).

 

If you just want to get the job done you could probably knock something up in Bash / Sed / Awk pretty easily. If your going to want to further develop and extend the functionality then Id'e recommend going with Python. If you've already experience in programming then you'll find Python pretty easy and very enjoyable to learn. It really is a great language.

 

PS: Pythons docs are just as good as php's IMO, you shouldn't really need any books.