I have a Perl program that is used as a postfix policy daemon - it basically does various checks on the sender's domain and decides whether to accept the message or discard it as spam. As we obviously get lots of messages from the same domains I'm putting the results into a hash tied to a BerkeleyDB - to speed lookups, share data between processes (there will typically be ~100 jobs running) and maintain the data over job restarts (postfix permits the policy jobs to run for a set period of time before restarting them).
All this generally works very well - I've seen it block up to 300,000 messages in one day and it will be stable for months, but occasionally one of the perl processes will balloon up in size until it consumes all the system memory. Eventually linux will kill the process and everything will be fine again. I've looked at my code a number of times and made some minor changes, but I still have this problem. I create the DB and tie with: 
my $env = new BerkeleyDB::Env -Flags => DB_CREATE | DB_INIT_CDB |DB_IN
+IT_MPOOL,
  -Home => '/etc/postfix/policydb';
my (%domains);
my $db = tie %domains,"BerkeleyDB::Hash",
  -Filename=> DOMAIN_FILE,
  -Flags=> DB_CREATE,
  -Mode => 0660,
  -Env=> $env
  or syslog 'error', "Can't open DB_File ". DOMAIN_FILE .": $!";

[download]
and the only place it is used is a lookup: 
  return $domains{$domain} if ($domains{$domain});

[download]
And an add if the domain wasn't found: 
  # Add to the berkeleydb unless some other process has already done i
+t
  unless($domains{$domain}) {
    my $lock=$db->cds_lock();
    $domains{$domain}=$return;
    $db->db_sync();
    $lock->cds_unlock();
  }

[download]
Any suggestions would be greatly appreciated.

In reply to Memory problems with BerkeleyDB by rjb

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.