I have a Perl application which has been running to my satisfaction for some five years. It uses MLDBM with DB_File and Storable to maintain a hash of arrays. This represents an index of words. Keys are single words; values are arrays of record numbers. From time to time, new words are added to the index. Recently, the addition of a new word caused overall performance to suddenly deteriorate. Output is unchanged, but speed is reduced by a factor 3. This change only occurs with certain, apparently unrelated, words. If the offending words are deleted, other words can be added as before with no effect on performance. The Windows system monitor shows an enormous increase in paging after one of the problem words is added, suggesting to me that the whole hash fitted in memory before the addition, but not after. However, the size of the MLDBM file on disk (336 KB) is almost unchanged. Moreover, inserting extra memory to give a 50% increase (512 => 768 MB) has no effect on the speed whatever. Suggestions? P.S. The index is updated as follows: 
my $arref = (exists $index->{$word}) ? $index->{$word} : []; # if word
+ in index, extract reference
    push(@{$arref}, $recno); # add the new record number to the refere
+nced array
    $index->{$word} = $arref; # put the reference back in the database

[download]

In reply to MLDBM performance problem by henrywalker23

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.