Maybe you can pre-process the log file then quickly display the results when queried. I have processed website access logs that were about 1GB daily. The reports were detailed and could be accessed in a few seconds, despite it taking several hours to produce them.

If your analysis is not I/O bound (or even if it is) you can prototype your analysis in Perl, then use profiling to identify the critical bits and improve them.