in reply to BerkeleyDB and a very large file

The retrieval rate for data in Berkeley DB files is very quick, and does not depend in any noticeable way on the amount of data in a given DB file. If the basic hash-lookup strategy is adequate to your needs, it's a good way to go.

But doing the initial creation of a really large Berkeley DB file may take a very long time for a data set in your range (100 million records), so have patience (and budget time) for that step.