Re: Reading binary files - program structure

Fairly self-documenting - e.g. using explicit variable names, which can then match the documentation for the file structure
For explicit naming, you'll have to either use my blocks, or hash keys, or use constant. For efficiency and compactness, using constant names defined to indexes in an array is a good option for largish numbers of discrete values:
```
use constant {
    THIS => 0, THAT => 1, THEOTHER => 2, ...
    TEMPL1 => 'N A10 S',
};

my @discrete = unpack TEMPL1, read( $file, $size );

print "THIS:", $discrete[ THIS ];
[download]
```
Compact - no great gobs of identical repeated read() and unpack() statements
For multi-dim arrays, use subroutines:
```
sub get2DArray {
    my( $x, $y, $templ, $templSize, $fh ) = @_;
    my @array;
    for my $y ( 0 .. $y - 1 ) {
        push @array, [ unpack $templ . $x, read( $fh, $templSize * $x 
+) ];
    }
    return \@array;
}

my $array2D = get2DArray( 100, 100, 'N', 4, $fh );
[download]
```
You could use nFor or Loops to write a generic multi-dim array reader, but unless you;re going above 3 or 4 dims, separate subs is probably easier. Watch the iteration order; it's can vary.
Fast - need to handle 20GB+ of data here.
On my system, using a combination of :perlio on the open & binmode gives me the best reading speed. See Re^2: Perl's poor disk IO performance for details. YMMV.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: Reading binary files - program structure Select or Download Code

Replies are listed 'Best First'.
Re^2: Reading binary files - program structure by ikegami (Patriarch) on May 20, 2010 at 15:51 UTC
An alternative is to your first snippet is to use a hash. `use constant TEMP1 => 'N A10 S'; my %discrete; @discrete{qw( this that theother )} = unpack TEMPL1, read( $file, $size ); print "this:", $discrete{ this };` [download] The constants are more typo resistant, though.	[reply] [d/l]
Re^3: Reading binary files - program structure by BrowserUk (Patriarch) on May 20, 2010 at 16:01 UTC
I did mention hashes along with my blocks of discrete named vars. The main thing I like about the constant method (besides the typo resistance which is good), is that simplicity of in order iteration. Of course you can get that by putting the hash keys into an array, but once you've done that, you're better off using the package stash rather than lexical hash. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply]
Re: Reading binary files - program structure by Anonymous Monk on May 24, 2010 at 17:16 UTC
Thanks for all the suggestions - now posting an update for anyone who stumbles on this. I started with something like almut's state machine but didn't like the way the structure of the file I was trying to read is then encoded into the script structure - feels like poor code/data separation. What I've ended up with is more like BrowserUk's suggestions. The crucial point (which I didn't really realise when I originally posted) was that the file format defines how many words should be read for each "variable" in the file, although in most cases that length is actually defined by algebra involving previously-read variables. So we can do something like this: my %v; # this hash holds all the data read # define the names of the "variables" to read from the file # * in the correct order * my @varnames = qw/this that theother/; # define where the length to read is not a default my %varlengths = ('that' => 2, 'theother' = '$v{this}*2'} foreach my $var (@varnames){ # get the length to read my $readlength; if(! exists($varlengths{$var}) $readlength = 1; # default } else { $readlength = eval($varlengths{$var}); # compute it } # read into %v if ($readlength == 1){ # read a scalar $v{$var} = unpack TEMPL1, read( $file, $readlength ); } else { # read into an anon. array $v{$var} = [ unpack TEMPL1, read( $file, $readlength ) ]; } } [download]	[reply] [d/l]
Re^2: Reading binary files - program structure by BrowserUk (Patriarch) on May 24, 2010 at 20:38 UTC
Re^3: Reading binary files - program structure by Anonymous Monk on May 25, 2010 at 16:47 UTC
Some notes below your chosen depth have not been shown here