I am working on a script to read and interpret a binary file, which consists of a variable number of variable length strings.
There are count values embedded in the file for the number of strings, and the length of each string, e.g (artificial example): 
 
NumberOfStrings (1 byte)
Str1Length (1 byte) Str1Characters (Str1Length bytes)
Str2Length (1 byte) Str2Characters (Str2Length bytes)
...
StrNLength (1 byte) StrNCharacters (StrNLength bytes)

[download]
The script itself contains something like this: 
# Create an example byte-sequence
$count = 3;
$str1 = "First string";
$str2 = "Second string";
$str3 = "Third string";
$bytes = pack("C C A* C A* C A*", $count, length($str1),
   $str1, length($str2), $str2, length($str3), $str3);

unpack("C", $count);
$bytes = substr($bytes, 1);
foreach $i (1 .. $count) {
   $length = unpack("C", $bytes);
   $str = unpack("xA$length", $bytes); # null byte skips length
   $bytes = substr($bytes, $length + 1);
   print "$str\n";
}

[download]
My question is - how can avoid the constant substr calls to remove the bytes I have already read?. I know I could use the "C/A*" template to read the length and string in one go, but that doesn't handle the variable number of strings.
In the real-life example, there may be millions of strings, and there may actually be other data types (ints) interspersed. I do actually read the binary file in chunks, so the actual number of bytes that substr is acting on is relatively small, but there must be a more efficient way? Can I have a pointer into the byte stream and call unpack on that? Maybe some of the modules that allow a file to be accessed as a scalar variable might help? Note: I did also try replacing the substr with a variant of the unpack, but it didn't seem to improve performance, and seemed to cause extra bytes to be consumed?: ($str, $bytes) = unpack("x C/A* A*), $bytes) Thanks!

In reply to Faster way to parse binary stream by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.