I have a script for loading a database where I need to keep a hash of the record ids as I prepare the files for loading. When loading a large number of records, the hash memory requirements are high enough that Perl crashes. To get around this issue, I've tied the memory hogging hash to disk using BerkeleyDB with the following code: 
use BerkeleyDB;
    
tie %HASH, "BerkeleyDB::Hash",
-Filename => "$tmpdir/hash.$$",
-Flags   => DB_CREATE,
    or die "Cannot open file $tmpdir/hash.$$: $! $BerkeleyDB::Error\n"
+;

[download]
When running with this, there is a very significant performance hit, much longer loading time than a memory based hash.

Is there another technique for getting around these memory limits, which might not have as high a performance hit?

In reply to Speeding a disk based hash by albert

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.