I would suggest not reinventing the wheel here. Analog is an excellent open source weblog analyzer. If you want prettier pictures for point-haired bosses then look to WebTrends.

Both of these products have been around for a while and are easy (very easy) to setup in *nix and Win32.

But to answer your question...
I think you are hitting the practical limitations of flat files. I would suggest going to a real database of some kind. PostgreSQL is an excellent database that will meet all you needs and more, and it runs in *nix and Win32(with some finagling). Sorry no Mac.

Don't reinvent good wheels

UPDATE: as trs80 pointed out it looks like I am off base with the Analog. I would still suggest looking into a database solution or trs80's solution of a series of flat files. If you think you will be getting a lot of traffic I would lean towards the DB.

grep

grep> cd pub grep> more beer