sf_ashley has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

This question is in a similar vein to that asked by Dwalin (#685248) and from how I gather seems a hot-topic for questions. I should also mention that I asked a very similar question in another forum but no response was given. Therefore, I hope it isn't too trivial a problem.

Anyway, I have written a script that outputs data to a binary format, via the rather simple statement.

open (OUTFILE, ">","$filename");
binmode OUTFILE, ":raw";
foreach $out (@data){
    print OUTFILE pack('V', $out);
}
close OUTFILE;

[download]

Now this file is manipulated by an old program but leaves it in the same low-endian form. What I would like to do is read back the binary file into a perl script and then further manipulate it. From a suggestion found in (#685248), I wrote the following test script:

#!/usr/bin/perl -w

$filename = "data.spc";
$recsize = 4;
$x = 1;
$i = 0;

open (FILE, "<", "$filename") or die "$!";

while (read(FILE, $buf, $recsize) == $recsize) {
  @values = unpack("V", $buf);
  @datapoint = splice(@values, 0, $x);
  $i += 1;
}

printf ("%d\n",$i); #No. of loops performed
unlink "temp.dat";
open (OUTFILE, ">", "data.dat");
foreach $datapoint (@datapoint){
    printf OUTFILE ("%d\n",$datapoint);
}
close OUTFILE;

[download]

The thing that has absolutely confused me is that while the number of iterated loops displayed is correct, but the data.dat file contains only a single number one, rather than 16,383 data points that it 'suggestively' looped over. Also, changing the value of $x did not seem to affect this result.

So does the fault lie in the unpacking of it, the splicing of it, or a very trivial error that I have not spotted? Also, in the unpack statement, I assume that the format indicates what the data should be read as.

Comments and suggestions on this matter would be very much appreciated.

Regards

Steve

Replies are listed 'Best First'.
Re: Simple query regarding unpack/splice
by moritz (Cardinal) on May 08, 2008 at 16:09 UTC
    @datapoint = splice(@values, 0, $x);

    So you take $x items from from @values, and assign it to @datapoint. How many values will @datapoint have at the end? Right, exactly $x. Which is 1, in your script.

    Update: BTW this is a bad idea: while (read(FILE, $buf, $recsize) == $recsize) {

    The last chunk of the file might be shorter than $recsize, in which case read returns a smaller number. The body of the loop is then skipped. If you want to read all records from your file, just say 

    while (read(FILE, $buf, $recsize)){
   ...
}

    [download]
[reply]
[d/l]
[select]
      moritz wrote:
      Update: BTW this is a bad idea: while (read(FILE, $buf, $recsize) == $recsize) {
      However, if you do get a short read, do you really want to perform the unpack?

      You can always detect a short read by checking the length of $buf at the end of the loop: 

      while (...) {
}
if (length($buf) > 0 && length($buf) < $recsize) {
  # last read was short
}

      [download]
      When reading from a disk file (that no one else is writing to), the only place you'll get a short read is at the end of the file. This isn't true for a pipe or a socket where there are more circumstances that will give you a short read before you've reached eof.
[reply]
[d/l]
[select]
      Indeed, it was evident that it was not iterating over @datapoint, so replacing @datapoint with $datapoint[$i] or push @datapoint, yielded a working script. My sincere thanks for your suggestions regarding this matter moritz.
[reply]
[d/l]
[select]
Re: Simple query regarding unpack/splice
by BrowserUk (Patriarch) on May 08, 2008 at 17:54 UTC

    Unless you're dealing with very large numbers of values, you're working way too hard. pack takes a list of inputs and can process a whole array at a time. So the output to a file can be as simple as: 

    open OUT, '>:raw', 'data.bin' or die $!;;
print OUT pack 'V*', @values;;
close OUT;;

    [download]

    And to read them back, unpack can process an unknown length string and produce a list. So that makes it as simple as: 

    open IN, '<:raw', 'data.bin' or diie $!;;
my @values = unpack 'V*', do{ local $/; <IN> };;
close IN;;

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
Re: Simple query regarding unpack/splice
by apl (Monsignor) on May 08, 2008 at 16:19 UTC
    I think you want to replace 
      @datapoint = splice(@values, 0, $x);

    [download]
    with 
      push @datapoint, splice(@values, 0, $x);

    [download]

    That is, you want to add a new entry to the datapoint array.

    While you're at it, you might want to

    • use strict; use warnings;
    • my your variables
    • test the error returns from your unlink and second open
    • use $i++; rather than $i += 1;
[reply]
[d/l]
[select]
      Thanks for the suggestion apl. Indeed, probably having the strict and warnings pragma enabled could well have highlighted this.
[reply]
Re: Simple query regarding unpack/splice
by citromatik (Curate) on May 08, 2008 at 16:09 UTC

    How many elements has @datapoints after the while loop? I don't know if there are more errors, but maybe one problem is in the line:

    @datapoint = splice(@values, 0, $x);

    [download]

    After the while loop @datapoint will only have the last value unpacked, you should use push @datapoint, splice(@values,0,$x) instead

    Also, taking into account that unpack called in scalar context returns only the first element unpacked, you can just simply write

    push @datapoint, unpack ("V",$buf)

    [download]

    Update (thanks to moritz):

    push @datapoint, scalar unpack ("V",$buf)

    [download]

    Hope this helps

    citromatik

[reply]
[d/l]
[select]
      But push provides list context onto its arguments.
[reply]
      Thanks for your suggestions regarding this matter citromatik. Indeed, by using push, the problem was fixed.
[reply]

Back to Seekers of Perl Wisdom