rjb has asked for the wisdom of the Perl Monks concerning the following question:

I have a Perl program that is used as a postfix policy daemon - it basically does various checks on the sender's domain and decides whether to accept the message or discard it as spam. As we obviously get lots of messages from the same domains I'm putting the results into a hash tied to a BerkeleyDB - to speed lookups, share data between processes (there will typically be ~100 jobs running) and maintain the data over job restarts (postfix permits the policy jobs to run for a set period of time before restarting them).
All this generally works very well - I've seen it block up to 300,000 messages in one day and it will be stable for months, but occasionally one of the perl processes will balloon up in size until it consumes all the system memory. Eventually linux will kill the process and everything will be fine again. I've looked at my code a number of times and made some minor changes, but I still have this problem. I create the DB and tie with:
my $env = new BerkeleyDB::Env -Flags => DB_CREATE | DB_INIT_CDB |DB_IN +IT_MPOOL, -Home => '/etc/postfix/policydb'; my (%domains); my $db = tie %domains,"BerkeleyDB::Hash", -Filename=> DOMAIN_FILE, -Flags=> DB_CREATE, -Mode => 0660, -Env=> $env or syslog 'error', "Can't open DB_File ". DOMAIN_FILE .": $!";
and the only place it is used is a lookup:
return $domains{$domain} if ($domains{$domain});
And an add if the domain wasn't found:
# Add to the berkeleydb unless some other process has already done i +t unless($domains{$domain}) { my $lock=$db->cds_lock(); $domains{$domain}=$return; $db->db_sync(); $lock->cds_unlock(); }
Any suggestions would be greatly appreciated.

Replies are listed 'Best First'.
Re: Memory problems with BerkeleyDB
by TOD (Friar) on Aug 19, 2007 at 01:06 UTC
    how large do the data usually get, let's say during one day? i mean, if it's a manageable amount you might be better off with shared memory and perhaps some sort of a periodical performed dumping process, which snychronizes the shmem data and the contents of the database.
    --------------------------------
    masses are the opiate for religion.
Re: Memory problems with BerkeleyDB
by Anonymous Monk on Aug 19, 2007 at 02:12 UTC
    see what sleepycat documentation suggests