in reply to combine multiple files into one (line by line)

Why are you openning and closing the same files over and over? This is expensive and error-prone. Just do something like this: 
my @files = ("1.txt", "2.txt", "3.txt");
my @fhs;

foreach my $file (@files) {
     open($fh, $file) || die;
     push(@fhs, $fh);
}
open(MIXED, "mixed.txt") || die;
while (1) {
     foreach $fh (@fhs) {
         $line = <$fh>;
         last if (eof($fh));
         print MIXED $line;
     }
}
map { close($_); } @fhs;
close(MIXED);

[download]
Note that this code assumes all your files have the same number of lines. If this is not the case, you have to do some extra trickery to not break out of the loop until all files are empty...

Replies are listed 'Best First'.
Re: Re: combine multiple files into one (line by line)
by Anonymous Monk on Apr 05, 2001 at 22:53 UTC
    Thank you for your help, however I failed to mention that the files range in size from 55MB to 130MB. In addition, I only want to grab the first 2,000 or so lines from each file and distribute them in the abovementioned fashion.

    For these reasons I don't think that using arrays is feasible. I understand that opening and closing is expensive, but hopefully less so than using huge arrays.

    Thanks for your help,

    Roger

    p.s. thabenksta's code works, but yours does not.

[reply]
      I don't think he was intending to load the entire file into memory. The trick is to keep an array of open FILEHANDLES and read one line at a time from each... 
      #!/usr/bin/perl -w
use strict;
use IO::File;

#interleave files, first argument is line count, rest are files.

my $lc = shift;
my @fhs;

foreach (@ARGV) {
  my $fh = new IO::File;
  open ($fh, "<$_");
  push @fhs, $fh;
}

while (--$lc) {
  foreach (@fhs) {
    print scalar(<$_>);
  }
}

      [download]

      Tested minimally but the path is clear from here I hope.

      --
      $you = new YOU;
      honk() if $you->love(perl)

[reply]
[d/l]
      You really don't need to read all files in. For clarity I will use filehandles. 
      use FileHandle;

my @files  = qw( 1.txt 2.txt 3.txt);
my @handles = map{ 
     $_ = new FileHandle $_, 'r';
     die "Could not open $_: $!" unless defined($_);
     $_;
   } @files;
open MIXED, ">$mixed_file" or die "Could not open $mixed_file: $!";

while( @handles ){
  foreach my $fh (@handles){
     my $line = $fh->getline or undef $fh;
     print MIXED $line if defined $fh;
  }
  @handles = grep defined, @handles;
}
close MIXED;

      [download]
      Hope this helps,

      Jeroen
      "We are not alone"(FZ)

[reply]
[d/l]
      Whoops, that's what happens when you submit code without testing it first :). There were four things wrong with my code:
      1. The open call for the MIXED file did not have a '>'. This is sheer carelessness.
      2. I was openning the files to the same typeglob, which resulted in three references to the same open file being stored in the array. This is a more subtle bug.
      3. I was testing for eof after reading in a line, instead of before. This would result in the last line never being written to the mixed file.
      4. Finally, I was using 'last' within the inner loop, foolishly thinking it would get me out of the outer loop. Another dumb mistake.
      Working code (at least on my Wintel box) follows: 
      my @files = ("1.txt", "2.txt", "3.txt");
my @fhs;

for ($i = 0; $i < scalar(@files); ++$i) {
     open($fhs[$i], $files[$i]) || die;
}
open(MIXED, ">mixed.txt") || die;
OUTER:
while (1) {
     foreach $fh (@fhs) {
         last OUTER if (eof($fh));
         $line = <$fh>;
         print MIXED $line;
     }
}
map { close($_); } @fhs;
close(MIXED);

      [download]
      Let this be a lesson for me in preliminary testing!

      -ton

[reply]
[d/l]

In Section Seekers of Perl Wisdom