in reply to Speeding up large file processing

Historically, the solution to the type of problem that you're describing would be to write a small server application that would intelligently cache the files, and cause your application to become a client which would send queries for data to the small server application.

In the spirit of major code re-use, I suggest using something like MySQL or DBD::SQLite2 (a self-contained, lightweight, file-based database and driver).

<-radiant.matrix->
Larry Wall is Yoda: there is no try{} (ok, except in Perl6; way to ruin a joke, Larry! ;P)
The Code that can be seen is not the true Code