in reply to Re: DB_File/BerkeleyDB with large datafiles
in thread DB_File/BerkeleyDB with large datafiles
I really like that suggestion, and would encourage you to pursue it if you can. An SQLLite database has many of the advantages of a Berkeley DB-file, but it can easily store more than one field. There might be size limits, however.
Another thought is ... what if you stored the data as individual files, in a pre-determined directory, then used (say) an SQLLite database file simply to index them? You query in order to find the names of the files that contain the information you want. Then, you open the individual files to get it.
File systems are, in effect, “databases” that are well-optimized to store gigabytes of data under millions of keys...