I think that we are guessing because we don't know the size of the files. I ran a quickie test of generating 10 million hash keys on my machine and on another machine. Results are below. I suspect that some idea of sorting the files with command line utilities and/or using a hash table approach will work out fine and that a DB isn't needed. Heck just keeping one file in memory may be enough! The initial algorithm timing would just skyrocket with files on the size of 10 million lines! I mean if both files are 10 million lines, reading DATA2 10 meg times and parsing it each time is gonna take a while! I don't think that the OP's files are that big, given that he can actually get a result in a few days.

I figure that something far less complex than a DB will work out just fine once the huge order of magnitude problems with the algorithm are addressed. 

#!/usr/bin/perl -w
use strict;
use Benchmark;

timethese (1,
  { bighashcreate => 
     q{
         my %hash;
         my $max_key = 10000000; #10,000,000: 10 million hash keys
         for (my $i =1; $i<=$max_key; $i++)
         {
             $hash{$i}=1;
         }
      },  
  },    
);
__END__
On my wimpy Prescott class machine on Windows XP:
Benchmark: timing 1 iterations of bighashcreate,
bighashcreate: 81 wallclock secs (78.03 usr +  1.84 sys = 79.87 CPU) 
               @  0.01/s (n=1)

On a server class machine under Linux (running as an average user):
Benchmark: timing 1 iterations of bighashcreate...
bighashcreate: 23 wallclock secs (22.21 usr +  0.91 sys = 23.12 CPU) 
               @  0.04/s (n=1)

[download]

In reply to Re^4: quicker way to merge files? by Marshall
in thread quicker way to merge files? by nessundorma

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.