in reply to How to improve speed of reading big files

Reading files a line at a time costs you something. Even though filehandle processing is buffered, you are still paying the price for re-filling the buffer at irregular intervals (based on when a line spans a buffer read). If you switch to IO::Handle and use the read method to suck in the entire file first, it will probably be faster. You will, of course, need enough RAM to hold the whole file while you process it. If this is still not fast enough, IO::Handle usually includes the setvbuf method, which allows you to increase the read buffer size. Increasing this to something like 256K may also improve this portion.

Replies are listed 'Best First'.
Re^2: How to improve speed of reading big files
by almut (Canon) on Sep 17, 2009 at 17:18 UTC
    IO::Handle usually includes the setvbuf method

    Unfortunately, since the advent of PerlIO, setvbuf() is no longer available... (current perls use a fixed-size 4k buffer) — see also 4k read buffer is too small.