Given what you need to do, I think you may find you're better off separating the log parsing and data analysis phases. Have one program who's sole function is to parse the log files and dump the results into a database, and a second to process the data in that database. (You could alter the system to log directly to a DB rather than a file, but I wouldn't recommend that for reliability reasons)

Doing this has the following advantages:

• You can offload the processing to a separate machine or machines
• You have access to a wider range of data without having to parse the log files multiple times
• You can write a variety of reports much more easily
• You can separate the DB server and report machines as well if you're finding you have bottlenecks there.

It'll make your life a lot easier in the long run, and give you access to more sophisticated data analysis tools. Setting up MySQL or Postgres isn't that big a deal--I'd really recommend you give it a shot this way.