in reply to How to improve speed of reading big files

I had a similar problem earlier where I had to speed up the text file parsing part of my application.

Without really going through your code, here are a few tips that may or may not come handy.

  • I recommend Devel::NYTProf for profiling. It gives you a very nice color-coded, line-by-line report on your code that you can use to figure out exactly where the bottlenecks are.

  • I found that substr is the fastest way to parse data out of lines that consist of fixed width fields only.

  • tr/// is faster than s///. (I don't know whether this is applicable here, though.) Anyway, the fewer regexps, the better.

  • In the inner conditional of your filterLog function, do you really need to test the existence of your $opts{} variables?
    • Comment on Re: How to improve speed of reading big files