julio_514:

A simple variation of a merge sort would do the trick. I did a node a while back with code to do it. You'd just need to change the part where it merges to compare and dump the value in the appropriate output files. A brief (untested) example:

#!/usr/bin/perl
use strict;
use warnings;

my ($infile1, $infile2) = ('abc', 'def');
my ($outfile1, $outfile2, $outfile3) = ('out.both', 'out.1', 'out.2');

open my $IF1, "sort -k3 $infile1 |' or die "opening $infile1: $!";
open my $IF2, 'sort -k3 $infile2 |' or die "opening $infile2: $!";
open my $OF1, '>', $outfile1 or die "opening $file1: $!";
open my $OF1, '>', $outfile2 or die "opening $file2: $!";
open my $OF1, '>', $outfile3 or die "opening $file3: $!";

my $f1 = <$IF1>;
my $f2 = <$IF2>;

while (defined($f1) or defined($f2)) {
    if ($f1 eq $f2) {
        print $OF1 $f1;
        $f1 = <$IF1>;
        $f2 = <$IF2>;
    }
    elsif ($f1 lt $f2) {
        print $OF2 $f1;
        $f1 = <$IF1>;
    }
    else {
        print $OF3 $f2;
        $f2 = <$IF2>;
    }
}
[download]

Also, if you plan on doing many operations like this, you may just want to dump your data in a database, and let the database do the work.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

my %hash;
open file1...
while (<$file1>)
{
  ... do something to get the id
  $hash{$id}++;  
}
close file1
open file 2
while (<$file2>)
{
  ... do something to get the id
  $hash{$id}+=3;
}
close file 2
open 3 output files: 
    the both file, file1 only and file2 only
loop back thru file1 and file2
for each line decide where it goes
  if hash value of the id ==1 file one only
  if hash value of the id ==3 file two only
  if hash value of the id ==4 both files
  negate the hash value to signal that this id
  has already been dealt with - each id should only
  appear in one of the 3 output files.
[download]

I first made your two 1.5 million row files to test. The ID has a lenght of 38 characters and is guaranteed to be unique within each file. Some IDs however can be found in both files.

The following program reads both files and produces three arrays: @both (containing the IDs which were found in both files), @first and @second (IDs only found in one of both files).

use Modern::Perl;
my %first_file;
{
    open my $FIRST, '<', 'first.txt' or die $!;
    while (<$FIRST>) {
        chomp;
        $first_file{$_} = '' ;
    }
}
my (@both, @first, @second);
{
    open my $SECOND, '<', 'second.txt' or die $!;
    while (<$SECOND>) {
        chomp;
        if (exists $first_file{$_}) {
            push @both, $_;
            delete $first_file{$_};
        }
        else {
            push @second, $_;
        }
    }
}
@first = keys %first_file;
[download]

With memory sizes as they are today, 1.5 million rows is “just not that interesting.”   If you had data volumes several times larger than that, you could do it like they did it before computers came along:   sort both trays of punched-cards into an identically sorted sequence, then run both trays at the same time through the merge-machine.

If you are using an SQL database, simply ORDER BY the same key.   When you process n data streams that you know to be identically sorted, you simply select the smallest of the “current records” from each of the n data streams until all streams are at end-of-file.

External sorting algorithms are among the most heavily studied.   Dr. Knuth called one of his tomes, Sorting and Searching, for a very good reason.

Guys, thank you so much for your suggestions! Using hash tables works like a charm. I won't post my original script because I would be embarrassed :|

I apologize, the post title is misleading - it is true that these are not huge files by today's standards.