in reply to Speeding up large file processing
Historically, the solution to the type of problem that you're describing would be to write a small server application that would intelligently cache the files, and cause your application to become a client which would send queries for data to the small server application.
In the spirit of major code re-use, I suggest using something like MySQL or DBD::SQLite2 (a self-contained, lightweight, file-based database and driver).
|
|---|