Update 29/08/2008:
Using vec() as suggested by BrowserUk below works fine - but unfortunately not fast enough for matching +500Mb file-1 with a one million lines in file-2.
It took more than one hour of processing time *sigh*
In the end, sorted file-2 then use File::SortedSeek works much faster.

Hi fellow monks,

I'm trying to find a best solution to match data between 2 huge files. The content of the files will be like this: 

file-1:                     file-2
1000 - foo                  2000
2000 - bar                  3000
3000 - fubar                .... 
...                         990000 
1000000 - heck

[download]

And the code should just print out everything in file-1 that is matched with the Id in file-2.

An example of solution would be:

A linear solution: 
use autodie qw(open close);

open my $fip, '<', 'file-1';
open my $fop, '<', 'file-2';

LINE1:
while (my $line1 = <$fip>) {
    my @token = split(/-/, $line1, 2);

    while (my $line2 = <$fop>) {
        chomp $line2;
        if ($token[0] == $line2) {
            print $line1;
            next LINE1;
        }
    }
}

close $fip; 
close $fop;

[download]

Which works fine, except toooooo slow for comparing more than a million line in both files as it needs to redo the inner looping again and again.

Another solution would be to use a hash table to store all Id in file-2 and check if $hash{$token[0]) exists, e.g.: 

use autodie qw(open close);

open my $fip, '<', 'file-1';
open my $fop, '<', 'file-2';

my %record_id;
while (my $line = <$fop>) {
   chomp $line;
   $record_id{$line} = 1;
}

close $fop;

while (my $line = <$fip>) {
    my @token = split(/-/, $line, 2);
    if ( exist $record_id{$token[0]} ) {
        print $line;
    }
}

close $fip;

[download]

Which works really fast, but then I don't want to load a million lines into memory at once.

What would be the best solution/approach in that situation?
Any CPAN module available that I can use out of the box?

Thanks,
Eddy

In reply to Matching data between huge files by est

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.