in reply to The Never-ending Database

I think arhuman definitly has the right idea. If you are looking for speed and want to save memory by not loading the entire file into memory, saving an index and using seek() is definitly going to be a good bet. What you may want to do is save the database file sorted by the primary keys, and use the Binary Search Algorithm's from Mastering Algorithms with Perl, this could reduce the amount of data in the index by allowing faster searches through the data.

Another thing to keep in mind is some sort of query optimization. By this i mean requests need to be screened to see if the will require a complete table scan, and in the case it may be worth reading in the entire file (rather than line-by-line with a seek, you could make it one huge sequential IO operation). Well, not too much new in this, but just some food for thought.


may the foo be with you