I just saw this on http://blogs.perl.org/users/makoto_nozaki/2015/07/grant-idea---pack-and-unpack-on-streams.html.

I wrote something like this where I had a template which described the format for the data stream.

My data stream was always like this so it was fairly easy to parse out

  [namespace][object_key][payload length][payload]
[download]

  namespace - a single character that allowed expansion in the future we only used one namespace.
  object_key - a single character that defined the object, payload length, and payload format
  payload length - was a function of object key but defaulted to unsigned 8 bit int (could be 16 bit or 32 bit int)
  payload - was a binary blob kind of like protocol buffers binary stream

So, based on which namespace and object key the payload was read (to length) and then passed to the correct payload formatter like this.

sub format {
  return [
           [Version1               => [1, 'C',      '%u', undef, 'chec
+kString']],
           [Version1String         => [0, 'a*',     '%s', undef, 'chom
+pString']],
           [Version2               => [1, 'C',      '%u', undef, 'chec
+kString']],
           [Version2String         => [0, 'a*',     '%s', undef, 'chom
+pString']],
         ];
}
[download]

  format is list "key" => "plan"
  plan is "length", "unpack", "sprintf display", "scale formula", "caller for extra work" (The plan should have been a hash not an array but I never got there.)

While reading, the format is actually shifted for each data element read and then passed by reference into the reader so that the reader can modify subsequent formats. In the example above the checkString function searches the read ahead buffer until it see a "\000" and then set the length on the next picture. (For new development recommend "string length" + "string")

The scale is this data [$a, $b, $c, $d] and the following formula $val=($a + $b * $val)/($c + $d * $val) which allows anything to be scaled quite easily. (not my formula but I cannot find source anymore)

Other examples

  ['Speed'                         => [2, 'v', '%u', [  0,1,100,0]]],#
+speed x 100 , 16 bit vax unsigned, 0 to 65535 (655.35) m/s
  ['Voltage'                       => [1, 'C', '%u', [200,1,100,0]]],#
+Voltage x100, + 2V, 8 bit char unsigned, 0 (2V) to 255 (4.55V)
  ['Method'                        => undef                        ],#
+0 length item for using the format for other things like building hum
+an displays
[download]

sub format {
  return [
           [ID           => [10, 'C[10]', '%02X%02X%02X%02X%02X%02X%02
+X%02X%02X%02X']],
           [MajorVersion => [1,  'C',     '%u']],
           [MinorVersion => [1,  'C',     '%u']],
           [SubVersion   => [1,  'C',     '%u']],
         ];
}
[download]

This was a great start and works quite well but, it only allows reading. I also needed to write the data back and that was not possible with this format structure.