Re: DBM::Deep overhead

Which approach is more appropriate for what you want to do? Purpose-wise, I submit that these two packages are apples and oranges.

Storable is a package that is designed for use as a persistence-framework. You take a moderate amount of data in RAM and “freeze” it into a representation that you can, say, stuff into an HTTP session-store.

DBM::Deep, on the other hand, describes itself as “a unique flat-file database module, written in pure Perl.” DBM::Deep::Cookbook, in the Performance section, makes the following additional caution:

Because DBM::Deep is a concurrent datastore, every change is flushed to disk immediately and every read goes to disk. This means that DBM::Deep functions at the speed of disk (generally 10-20ms) vs. the speed of RAM (generally 50-70ns), or at least 150-200x slower than the comparable in-memory datastructure in Perl.

Okay, nothing at all wrong with that. “This is what you get, and this is the price you will pay to get it.”

Consider all of your various options, even within DBM::Deep. For example, SQLite is a rock-solid single file on-disk data store that is used by everything on the planet including the cell-phone in your pocket. (It, too, normally commits all writes to disk immediately, so you must use Transactions effectively to get decent performance out of it.)

Replies are listed 'Best First'.
Re^2: DBM::Deep overhead by perlpipe (Acolyte) on Apr 21, 2011 at 02:21 UTC
Thanks for the distinction between Storable and DBM::Deep. I guess I was thinking more in terms of what Storable does than an actual database, or specifically one that is accessed by multiple programs/users concurrently. I was looking at it as an easy way of storing complex data so it could be retrieved later, one piece at a time. My observation on overhead wasn't just in terms of cpu but also the significant difference in file size. Will increasing the input data size from 36 K to 360 K result in a database size increase from 4 megs to 40 megs? But I assume there is an initial overhead that won't be changed much by an increase in number of records or an increase in record size. Thanks for the information on perl databases. It's something I am going to have to dig deeper into.	[reply]

Replies are listed 'Best First'.

Re^2: DBM::Deep overhead
by perlpipe (Acolyte) on Apr 21, 2011 at 02:21 UTC

Thanks for the distinction between Storable and DBM::Deep. I guess I was thinking more in terms of what Storable does than an actual database, or specifically one that is accessed by multiple programs/users concurrently. I was looking at it as an easy way of storing complex data so it could be retrieved later, one piece at a time.

My observation on overhead wasn't just in terms of cpu but also the significant difference in file size. Will increasing the input data size from 36 K to 360 K result in a database size increase from 4 megs to 40 megs? But I assume there is an initial overhead that won't be changed much by an increase in number of records or an increase in record size.

Thanks for the information on perl databases. It's something I am going to have to dig deeper into.

[reply]