You could save some memory consumption (and perhaps reduce slowdown due to memory paging) if you opened the files one by one. By opening all the files at once, you are also opening 200 buffers for each of the files all at once.

1. Create an array of arrays.
2. Read the first file one line at a time, and place all of column 2's values in $aa[0];
3. Read the second file one line at a time, and place all of column 2's values in $aa[1];
4. And so on...
5. Loop through array and print to file

With files that large, you might also want to consider tying the arrays holding the columns to random access files. I once had a ten-fold speed up (in a C++ program) just by using temp files instead of RAM to store data while I was processing. See Tie::File

If your columns are fixed width, you might be able to to avoid two cycles, one to read the array and one to print the array, by keeping a variable that stores the current line length. Then instead of an array of arrays, you could just:

1. Read the first file
1. Write the first file by placing each col 2 value on a separate line.
2. Increase the line length by one column width
2. Read the next file
   1. Seek to the end of the line, insert the column
   2. Seek to end of the column you just wrote
   3. Seek to end of next line, insert the column
   4. After all lines have been read, increase line length by one column width
3. Repeat until all files are processed

Best, beth

Hi,

Try CPAN module Tie::File
It says:

Tie::File represents a regular text file as a Perl array. Each element in the array corresponds to a record in the file. The first line of the file is element 0 of the array; the second line is element 1, and so on.

The file is not loaded into memory, so this will work even for gigantic files.

Changes to the array are reflected in the file immediately.

Cheers !

--VC

I used to use paste and cut. paste file1 file2 ...|cut -f2,3 > newfile.txt and actually it works. The timing with that approach is not that bad ~ 30 min or so for 200 files. However, I wanted to be more creative and use my Perl skills. Thank you very much anyway.

I assume you've tried it and discovered it's too slow ?

200 files at 6.9M lines each, each line three fields tab delimited -- if that's 60 bytes per line, you're reading ~80G bytes and writing ~27G (assuming roughly equal field sizes). On my little Linux box, I just timed cp foo bar for an ~13G file and it took ~15mins -- so 80G + 27G looks like about two hours work ? Of course, with faster drives and what-not, you may do better.

I've never tried opening 200 files at once...

It's hard to say what the processing time will add to this. It looks pretty I/O bound. If processing time is an issue, then I'd look at reading the files a chunk at a time... but I'd have to be convinced there was a problem; and even then I'm not sure that processing chunks of the files in Perl would be quicker than using Perl to read a line at a time.

So... what are you expecting, and what do you get when you try the straightforward approach ?

Wishing to squeeze as much as possible out of the inner loop, I think the following may be faster:

  my $atleastone = 1;
  while ($atleastone) {
    $atleastone = 0 ;

    my @l = () ;
    foreach my $op (@handles) {
      my $c2 ;
      ++$atleastone and (undef, $c2) = split if defined($_ = <$op>) ; 
+  
      push @l, $c2 || "0" ;
    } ;
    print OUTFILE join("\t", @l), "\n" ;
  } ;
[download]

  my $atleastone = 1;
  while ($atleastone) {
    $atleastone = 0 ;

    my $l = '' ;
    foreach my $op (@handles) {
      my $c2 ;
      ++$atleastone and (undef, $c2) = split if defined($_ = <$op>) ; 
+  
      $l .= ($c2 || "0") . "\t" ;
    } ;
    chop $l ; # Discard trailing "\t"
    print OUTFILE "$l\n" ;
  } ;
[download]

  my $atleastone = 1;
  while (defined($atleastone)) {
    $atleastone = undef ;

    my $l = '' ;
    foreach my $op (@handles) {
      my $c2 ;
      ($atleastone, $c2) = split if defined($_ = <$op>) ;   
      $l .= ($c2 || "0") . "\t" ;
    } ;
    chop $l ; # Discard trailing "\t"
    print OUTFILE "$l\n" ;
  } ;
[download]