Well really I don't think 120,00 is all that big. I recently wrote a script to parse out duplicates on our backup tapes. I clocked the number of cycles and it pushed near on 9 million in about 30 secs (from memory). And I'm not on on a ripper of a machine. Your 120,000 it should rip through relatively quickly.

Putting speed aside now I'm more interested in how you are weeding out duplicates. Using hashes will be faster. Use the email address as the key to the hash and just test if the key already exists. If it does then you know you have a duplicate and output to a seperate file or just ignore it completely.

I'll assume you know how to read in your emails into an array. Just do a foreach on the array and put each into the hash and then output any that already exists. For example below see below.

Enjoy!
Dean 
foreach (@array) {
    if (! $hash{$_}) {
        $hash{$_} = $_;
    } else {
        ## duplicates output here or ignore
        print $_;  
    }
}

# then print your hash to get your none duplicate results

[download]

In reply to Re: Removing duplicates in large files by crabbdean
in thread Removing duplicates in large files by TIURIC

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.