Sounds like an on-disk database is the way to go here.

In particular, take a look at the rather amazing DBD::SQLite, a complete SQL92 fast database with transactions and triggers... without the database server!

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.