The large tables are generated with map and grep loops. Basically making many combinations of a few things. I guess I will try dumping the static values in a flat file and DBM file, then benchmark it both ways. I guess my question now "is opening and reading a flat file faster that opening and reading a DBM file. Is there any performance benefits with one or the other?" | [reply] |
YMMV, depending on which flavor of DBM you pick and how your hardware gets along with it. As a general rule, writing to any sort of DBM file tends to be somewhat more expensive than writing a flat file, in terms of overall space consumed, amount of actual disk i/o performed, and total cpu time required.
But when reading data back after you've stored it, a DBM file is vastly better, especially when fetching values in a quasi-random fashion from a very large set -- or at least, whenever the fetching order is very different from the storage order. In such cases, doing repeated sequential searches over a flat file will kill you, whereas the DBM file is really just a big hash array on disk, optimized to deliver any chosen piece of data in a consistently short amount of time.
So the question really is "what sort of access do you really need when reading the data back?" If you can easily write a flat file such that you just need to read it back once from beginning to end, then a flat file will be the better choice.
| [reply] |