tanger007 has asked for the wisdom of the Perl Monks concerning the following question:

Been trying to merge a bunch of sorted files with the same ordering. The number of files is random, so I chose to use an array of file handles. The plan is to have each file send in one line, then, the line with the highest rank will get pushed out, and then another line is yanked out of the file where that last line was from... How does this work with perl? Been trying 3 hours with no success...
  • Comment on Merge huge files (individually sorted) by order

Replies are listed 'Best First'.
Re: Merge huge files (individually sorted) by order
by Loops (Curate) on Jul 18, 2013 at 23:57 UTC
[reply]
      Works so well I felt more stupid :) A follow up question: if you have a big file (>10GB) in which one column has say 100 unique values. How do you break this file into 100 smaller files with one unique value in that column? Thanks so much.
[reply]

        tanger007:

        Try something like putting a file handle for each column value in a hash, and then looking up the file handle on demand:

        my %OFH;
my $OFH;
while (<$IFH>) {
    my @fields = split /\t/,$_;
    $OFH = $OFH{$fields[$key_column]};
    if (! defined $OFH) {
        # We don't have this value yet, so open another file
        open $OFH, '>', 'key_value.' . $fields[$key_column];
        $OFH{$fields[$key_column]} = $OFH;
    }
    print $OFH join("\t",@fields);
}

        [download]

        Note: It's rough, untested and needs some error handling and such. But the basic concept should work fine for you.

        ...roboticus

        When your only tool is a hammer, all problems look like your thumb.

[reply]
[d/l]

Back to Seekers of Perl Wisdom