in reply to How to deal with Huge data

You could use DBD::SQLite to store and process the data and possibly "sqlite3" itself to load the data.