This sounds like a job for a database. Have you taken a look at DBD::CSVor DBD::RAM? I think one of those solutions would be easier to implement and certainly scale a lot better.