Re:^4 Binary file handling

I actually did something very similar in here. And tested with 6000*2000 matrix. Here's the code snippet:

open (INM,"$in_matrix") or die "Unable to open 'in_matrix': $!";
my @row=split (/\t/,readline(INM));
my @colfiles=();
my $max=$#row;
foreach (0 .. $max) {
  open ($colfiles[$_],">$out_matrix.$_") or die "Failed on opening tem
+pfile $_: $!";
  print {$colfiles[$_]} shift (@row);
}
[download]

Message was

Failed on opening tempfile 1019: Too many open files at ../bin/transposematrix.pl line 62, <INM> line 1.

So is that error from perl or from shell i'm executing the script? I think its from perl, since it tells me the line (correctly) from the script. So it seems to me that i need to do this in 1000 file groups if i do it this way.

Another thing which I though about this morning is combining your original way and idea of splitting the matrix. Now if i split it into 4 pieces, each piece needs to transposed 180 degrees, in which case your original code would do the trick. I figure it should work, even if the split isn't exact (eg. original matrix has 183 rows and 235 columns).

Comment on Re:^4 Binary file handling Download Code

Replies are listed 'Best First'.
Re: Re:^4 Binary file handling by tachyon (Chancellor) on Mar 19, 2004 at 13:04 UTC
It is Perl reporting an OS message. Consided open F, $foo or die $! say you get permission denied, that is an OS message delivered via perl. You don't have to use real file descriptors. You could simply open a DBM database or a tied hash, pretend that the keys are the file descriptors and just append to the values. It actually simplifies the code, but it will be a big speed hit. Here is one of the examples from Re: Binary file handling converted to use a hash tied to a file. sub rotate_minus90 { my ( $infile, $outfile, $tmp ) = @_; $tmp \|\|= 'c:/tmp/temp'; open IN, $infile or die "Can't read $infile $!\n"; # find number of columns and open a temp file for each chomp( local $_ = <IN> ); my @data = split ' '; my $num_cols = $#data; dbmopen(my %fhs, $tmp, 0666) or die "dbmopen can't grok $tmp $!\n" +; $fhs{$_} = "$data[$_]\t" for 0..$num_cols; while( <IN> ) { chomp; @data = split ' '; $fhs{$_} .= "$data[$_]\t" for 0..$num_cols; } close IN; open OUT, ">$outfile" or die "Can't write $outfile $!\n"; for ( reverse 0.. $num_cols ) { print OUT $fhs{$_}, "\n"; } dbmclose(%fhs); close OUT; } [download] cheers tachyon	[reply] [d/l]

Replies are listed 'Best First'.

Re: Re:^4 Binary file handling
by tachyon (Chancellor) on Mar 19, 2004 at 13:04 UTC

It is Perl reporting an OS message. Consided open F, $foo or die $! say you get permission denied, that is an OS message delivered via perl.

You don't have to use real file descriptors. You could simply open a DBM database or a tied hash, pretend that the keys are the file descriptors and just append to the values. It actually simplifies the code, but it will be a big speed hit.

Here is one of the examples from Re: Binary file handling converted to use a hash tied to a file.

sub rotate_minus90 {
    my ( $infile, $outfile, $tmp ) = @_;
    $tmp ||= 'c:/tmp/temp';
    open IN, $infile or die "Can't read $infile $!\n";
    # find number of columns and open a temp file for each
    chomp( local $_ = <IN> );
    my @data = split ' ';
    my $num_cols = $#data;
    dbmopen(my %fhs, $tmp, 0666) or die "dbmopen can't grok $tmp $!\n"
+;
    $fhs{$_} = "$data[$_]\t" for 0..$num_cols;
    while( <IN> ) {
        chomp;
        @data = split ' ';
        $fhs{$_} .= "$data[$_]\t" for 0..$num_cols;
    }
    close IN;
    open OUT, ">$outfile" or die "Can't write $outfile $!\n";
    for ( reverse 0.. $num_cols ) {
        print OUT $fhs{$_}, "\n";
    }
    dbmclose(%fhs);
    close OUT;
}
[download]

cheers

tachyon

[reply]
[d/l]