Both a database and a hash will do the trick. Which one is better for your application depends on a lot of things. Up to a certain number of keys, a hash will be faster. But if the number of keys grows, Perl will use more and more memory, and at a certain point, it will start trashing, and eventually run out of memory. Somewhere along the way, it's going to be more efficient to use a database, specially if the database is on another box. But where that point is depends on the number of keys, the number of inserts, the number of queries, and the amount of memory your program has available.

It's something only you can test.

In reply to Re: How to remove duplicates from a large set of keys by Anonymous Monk
in thread How to remove duplicates from a large set of keys by nite_man

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.