in reply to Re: Table Generation vs. Flat File vs. DBM
in thread Table Generation vs. Flat File vs. DBM

The large tables are generated with map and grep loops. Basically making many combinations of a few things. I guess I will try dumping the static values in a flat file and DBM file, then benchmark it both ways. I guess my question now "is opening and reading a flat file faster that opening and reading a DBM file. Is there any performance benefits with one or the other?"
  • Comment on Re: Re: Table Generation vs. Flat File vs. DBM

Replies are listed 'Best First'.
Re: Re: Re: Table Generation vs. Flat File vs. DBM
by graff (Chancellor) on May 05, 2004 at 07:01 UTC
    YMMV, depending on which flavor of DBM you pick and how your hardware gets along with it. As a general rule, writing to any sort of DBM file tends to be somewhat more expensive than writing a flat file, in terms of overall space consumed, amount of actual disk i/o performed, and total cpu time required.

    But when reading data back after you've stored it, a DBM file is vastly better, especially when fetching values in a quasi-random fashion from a very large set -- or at least, whenever the fetching order is very different from the storage order. In such cases, doing repeated sequential searches over a flat file will kill you, whereas the DBM file is really just a big hash array on disk, optimized to deliver any chosen piece of data in a consistently short amount of time.

    So the question really is "what sort of access do you really need when reading the data back?" If you can easily write a flat file such that you just need to read it back once from beginning to end, then a flat file will be the better choice.