Um, wouldn't that be faster by just inserting into a hash - when you get a new element, see if it's already in the hash?

However, the OP is really asking much more about SQL than perl. Not that it's off-topic in my opinion, but answers should probably focus on the OP's problem area. Lists and hashes are unlikely to be the best solutions.

The problem really is in defining what "duplicate" is. And then devising either tables or SQL queries (or some combination of both) to expose those duplicates. Even using lists or hashes, we still would need a better definition of what "duplicate" means to know what to sort on, or what to use as the hash key.

In reply to Re^2: redundancy Checker by Tanktalus
in thread redundancy Checker by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.