I had a similar problem 2 years ago. My solution was to cook up a weird disk-based hashing method. Each of my records had a primary identifier (a unique name) which I converted to ascii and summed the integers. I then multiplied by a really large prime number (for fun and further diversity) and then I added the ascii values for the first and last characters again and then did modulo division with 65536. The result for each record was a pretty quickly computed bucket for each record which was nearly random. I converted the remainder to hex so I'd get filenames between 00/00.txt .. FF/FF.txt ... up to 256 directories each with up to 256 text files. I added many hundreds of thousands of records and it was still wonderfully fast but it would have been horrendous if they'd all resided in a single file. If you don't have a unique key for each record, this solution would not be helpful but it werked really well for me back in the day.

Since then, I've picked up the O'Reilly MySQL & mSQL book and have lerned DBI. If you have permissions to setup a daemon on a server, it's really pretty painless. I've compiled MySQL a couple of times now and installed 3 or 4 different versions (at different times) without too much headache. SQL is like Perl in that it is a rich language but it's okay not to know everything immediately. There are a few key things that can be lerned quickly and you can get by with those... the rest just makes you better. SQL is easy enough and DBI seems to provide the portability and flexibility to plug in differing database engines and drivers rather simply. I've been won over to relational databases since they are really powerful and they're not as hard as I thought they'd be to use (maybe MySQL is a cakewalk compared to Oracle but it's a start). I hope this helps somewhat. I'd be glad to share more of my flatfile madness if you'd like. Shalom.

-PipTigger

p.s. Initiate Nail Removal Immediately!