In an ideal world, this would be the method I'd prefer to use also, though unfortunately we don't have the resources to do this at the moment.

Thanks for your input however, much appreciated.

This is false economy; depending on your skill level… Putting it all into SQLite is semi-trivial—Pg or MySQL only slightly harder—and would obviate the need for repetitive, selective, time consuming, and error prone reparsing.

Sometimes taking a time hit up front for extra code, structure, or learning will save 10 fold down the road.

I agree with Your Mother that it will be only a small effort (both in time, people, computer resources and costs) to put this data in a database. Parsing your log-files into an SQLite-database will not need more than 50 lines of Perl-script and SQLite needs next to no maintenance once it has been set-up. A MySQL-database, more robust and better scalable, needs more effort to set it up, but much of it is a one time cost which you may have already paid that for a large part if you have a MySQL server running somewhere in the company.

And if you are running an Apache web server, then there is always mod_log_mysql (although its installation is not for the faint-hearted it seems) which logs directly into a MySQL database.

CountZero

A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

My blog: Imperial Deltronics