Re: DB_File/BerkeleyDB with large datafiles

If DBD::SQLite is an option, even though it is not a real plain file, then Tie::DBD::Hash might be what you want. I wrote that because I had a very similar problem.

Enjoy, Have FUN! H.Merijn

Comment on Re: DB_File/BerkeleyDB with large datafiles

Replies are listed 'Best First'.
Re^2: DB_File/BerkeleyDB with large datafiles by locked_user sundialsvc4 (Abbot) on Sep 20, 2010 at 17:23 UTC
I really like that suggestion, and would encourage you to pursue it if you can. An SQLLite database has many of the advantages of a Berkeley DB-file, but it can easily store more than one field. There might be size limits, however. Another thought is ... what if you stored the data as individual files, in a pre-determined directory, then used (say) an SQLLite database file simply to index them? You query in order to find the names of the files that contain the information you want. Then, you open the individual files to get it. File systems are, in effect, “databases” that are well-optimized to store gigabytes of data under millions of keys...

Replies are listed 'Best First'.

Re^2: DB_File/BerkeleyDB with large datafiles
by locked_user sundialsvc4 (Abbot) on Sep 20, 2010 at 17:23 UTC

I really like that suggestion, and would encourage you to pursue it if you can. An SQLLite database has many of the advantages of a Berkeley DB-file, but it can easily store more than one field. There might be size limits, however.

Another thought is ... what if you stored the data as individual files, in a pre-determined directory, then used (say) an SQLLite database file simply to index them? You query in order to find the names of the files that contain the information you want. Then, you open the individual files to get it.

File systems are, in effect, “databases” that are well-optimized to store gigabytes of data under millions of keys...