I am working on a project that de-dupes MASSIVE numbers of addresses but I get an incredible slow down on my system after the first third run through. The files hold 10 million records a piece and there are up to 1 billion records within catagories. Obviously there is lot of duplication. The following code runs and runs correctly, but has some seeming limitations depending on platform. It is running on a Dual PIII with 1.5G RAM, W2k server. The RAM seems to fill and the slow to a crawl. Any suggestions?

I have tried to "undef" the array and that removes that RAM problem, but it still seems to slow down. (besides, if I did not "undef" the array, it would carry over and create some really massive files. The current file sizes average about 220Mb. 

foreach $key (sort keys %fcnt) {
   $fileNum = "001";
   while (-e "$key-$run-$fileNum.txt"){
      undef (%washOut);
      open (FILE, "<$key-$run-$fileNum.txt");
      print qq|Load/De-duping file (internal):\n|;
      while (<FILE>) {
         $tcnt++;
         $twash = $_;
         chomp($twash);
         unless ($washOut{$twash}){
            $washOut{$twash} = 1;
            $cnt++;
            if ($x >= 1000){ print qq|- $cnt\n|; $x = 0; }
            $x++;
         }
      }
      close(FILE);
      print qq|$tcnt Scanned\n$cnt Loaded.\nSave Deduped Info...\n|;
      open ($fileNum, ">$key-$run-$fileNum.new");
      foreach (keys (%washOut)){ print $fileNum qq|$_\n|; }
      close($fileNum);
      print qq|Renaming original .txt to .old file...\n|;
      rename ("$key-$run-$fileNum.txt", "$key-$run-$fileNum.old");
      print qq|Renaming extract .new file to .txt file...\n|;
      rename ("$key-$run-$fileNum.new", "$key-$run-$fileNum.txt");
      print qq|Deleting .old file...\n|;
      unlink ("$key-$run-$fileNum.old");
      print qq|Saved... Continuing...\n|;
      $fileNum++;
      $nextFile = $fileNum;
      # Wash against other files
      if (-e "$key-$run-$nextFile.txt"){
         $x = 0; $cnt = 0; $tcnt = 0;
         while (-e "$key-$run-$nextFile.txt"){
            open (FILE, "<$key-$run-$nextFile.txt");
            open (FILEout, ">>$key-$run-$nextFile.new");
            print qq|Loading and de-duping file (external):\n|;
            while (<FILE>){
               $tcnt++;
               $twash = $_;
               chomp($twash);
               unless ($washOut{$twash}){
                  print FILEout qq|$twash\n|;
                  $cnt++;
                  if ($x >= 100000){ print qq|-|; $x = 0; }
                  $x++;
               }
            }
            close(FILEout);
            close(FILE);
            print qq|Renaming original .txt to .old file...\n|;
            rename ("$key-$run-$nextFile.txt", "$key-$run-$nextFile.old");
            print qq|Renaming extract .new file to .txt file...\n|;
            rename ("$key-$run-$nextFile.new", "$key-$run-$nextFile.txt");
            print qq|Deleting .old file...\n|;
            unlink ("$key-$run-$nextFile.old");
            $nextFile++;
         }
      }
   }
}
Thanks in advance for the advice...
The Mage

In reply to Massive Sorting by The Mage

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.