Compression of a mini-database in memory

meonkeys has asked for the wisdom of the Perl Monks concerning the following question:

#!/usr/bin/perl -w
use Storable qw( freeze thaw );
use DB_File;
use strict;

my %hash = (
 1 => { desc   => 'fat',
        states => 'WA OR ID CA',
        open   => '0',
        narm   => '1',
      },
 2 => { desc   => 'fatter',
        states => 'NATIONWIDE OR ID CA',
        open   => '1',
        narm   => '0',
      },
);

my $frozen = freeze \%hash;

my %table;

tie (%table, 'DB_File', 'planners.db', O_RDWR|O_CREAT,
  0666, $DB_BTREE) or die "Cannot tie: $!";

%table = (
  planners => $frozen,
);

untie %table;
[download]

%table will be used to generate HTML pages throughout a website, and could grow to 5 key/value pairs. %hash would get up to a maximum of 200 key/value pairs. I want to keep %table alive in memory (I'm using mod_perl), and unfreeze/uncompress the values as I need them. Would Compress::Zlib work to keep %table's size in memory to a minimum? Is there another way that I can keep %table very small in memory, assuming CPU is not a bottleneck?

Comment on Compression of a mini-database in memory Download Code

Replies are listed 'Best First'.
Re: Compression of a mini-database in memory by AgentM (Curate) on Feb 17, 2001 at 02:33 UTC
Compress::LZV1, an extremely new module, may be exactly what you're looking for: compress and uncompress from within Perl. In fact, its blurb specifies: `extremely leight-weight Lev-Zimpel-Vogt compression.` [download] Whether this has anything to do with our own extremely is irrelevant. Enjoy! AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.	[reply] [d/l]
Re: Compression of a mini-database in memory by merlyn (Sage) on Feb 17, 2001 at 01:06 UTC
At this point, moving it into a real database would make more sense, perhaps with a LRU cache. -- Randal L. Schwartz, Perl hacker	[reply]
Re: Compression of a mini-database in memory by dws (Chancellor) on Feb 17, 2001 at 01:13 UTC
Can you say something about why you believe you need to keep `%table` "very small in memory"? 200 key/value pairs of the size you show above won't take up a significant amount of space (unless you're on a very, very constrained machine). Is there something about your constraints that you haven't mentioned, or wish to clarify?	[reply] [d/l]
Re: Re: Compression of a mini-database in memory by meonkeys (Chaplain) on Feb 17, 2001 at 01:21 UTC
My main concern is keeping the mod_perl process size down. However, after merlyn's response, I'll probably just use MySQL. I have no way to know what the size of %table is in memory.	[reply]
Re: Compression of a mini-database in memory by McD (Chaplain) on Feb 17, 2001 at 02:07 UTC
Am I missing something obvious? It strikes me that there's very little to be gained by compressing a data structure which is kept in memory. After all, the first time you uncompress it, your process eats more memory. In general, you can't give memory back to the operating system. Peace, -McD	[reply]
(tye)Re: Compression of a mini-database in memory by tye (Sage) on Feb 17, 2001 at 02:29 UTC
If you have N things and only ever keep 1 uncompressed in memory at a time, then you can reduce the memory required. - tye (but my friends call me "Tye")	[reply]

AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.