in reply to Re: Using pack to evaluate text strings as hexadecimal values
in thread Using pack to evaluate text strings as hexadecimal values

With apologies for resurrecting a dead horse (and mangling metaphors), pack/unpack give us the capability of going back and forth between strings and hex digits (H and h), and between strings and integers (i and n and friends), but not, as far as I can tell, between hex digits and integers, which I think was the original intent of davis. Yes, hex() and sprintf() let us do these, but there's no reason in principle why there couldn't be a pack/unpack format item to do the job.

Why bother, one might ask? I want to keep the "external form" of my records as pure ascii, so I can use all my favorite UNIX utilities on them. So how do I represent integer bit masks? I cannot use formats like n blindly, lest some bit mask results in a byte that looks like a newline or a null. I can "or in" 6 bits to 0x20 (ord(' ')), avoiding the high bit to stay ascii and the constant x20 bit, to avoid all the nasty control characters, but A) it's unlikely to port well to a non-ascii system and, more important, B) it's not at all easy to see what bits are on and what bits are off. The same applies to uuencoded masks using format u. I could represent the mask as a fixed width digit string, but that's B) still difficult to see what bits are on and off, and C) it takes a lot of characters to encode a few bits (5 digit numbers to accommodate 16 bits). I'd be happy to trade off the extra characters to represent the bits in hex, for the benefit of easy determination of what's on and what's off.

I can so this using hex() and sprintf(), but then I need a way to fiddle records before packing and after unpacking. Am I alone in wishing there were a pack format item for the conversion, so I could do the whole job with pack and unpack?

  • Comment on Re^2: Using pack to evaluate text strings as hexadecimal values

Replies are listed 'Best First'.
Re^3: Using pack to evaluate text strings as hexadecimal values
by BrowserUk (Patriarch) on Mar 21, 2011 at 18:26 UTC

    What doesn't this do that you'd like it to do? 

    $n = 0; $n |= 1 << $_ for 1,3,5,7;;
print $n, unpack 'H2', pack 'v', $n;;
170 aa

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

      It doesn't give me a single format string that I can use to convert back and forth between "records" and "fields".

        @fields = unpack($format, $record);
  $record = pack($format2, @fields);

      [download]

      (I actually use hash slices rather than arrays, but that's not germane here.) I don't mind having distinct formats. For all-alpha fields, I sometimes do

        ($format3 = $format) =~ s/A/a/g;

      so I can generate a non-trimming unpack format from the default, a convenience, but not absolutely essential, since setting up the formats is a one-time thing, but conversion is a per-record thing.

[reply]
[d/l]
[select]
        It doesn't give me a single format string that I can use to convert back and forth between "records" and "fields".

        The "problem" is that you cannot have a single conversion that produces binary numeric data for internal math use, and also produces asciified hex string for display purposes.

        It's the same as asking for a single printf format that will display an number as decimal and hex. It cannot be done.

        The solution is to have two templates: 1 for internal packing and unpacking: 

        $templ = 'A4 v d';;

@fields = ( 'fred', 12345, 123.456 );;

$record = pack $templ, @fields;;

print unpack $templ, $record;;
fred 12345 123.456

        [download]

        And another for unpack a record for display purposes:

        $templ2 = 'A4 H4 H16';;

print unpack $templ2, $record;;
fred 3930 77be9f1a2fdd5e40

        [download]

        There is simply no way to implement a single template that would server both purposes.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
Re^3: Using pack to evaluate text strings as hexadecimal values
by jpl (Monk) on Mar 28, 2011 at 14:04 UTC

    I'm going to "pop this up a few levels", because the exchange between BrowserUk and me isn't easily visible any more. Here's a (somewhat edited) version of the last couple exchanges. Sorry, my html smarts don't include making the quotations evident, and the Formatting Tips didn't help a lot.

    (jpl) Why couldn't we have a format item "Y" (where "Y" is some unused format character, finding which may be the real problem) such that 

      pack("Y4", 32);       # produces "0020"
  unpack("Y4", "0020"); # produces 32

    [download]

    There are many format characters, v among them, that pack integers into strings and unpack strings into integers. I believe the only practical difference between "Y" and "v" is the contents of the string. The one produced by "v" may be unsuitable for display, the one produced by "Y" would be both displayable and, when displayed, indicative of the bits in the integer from which it was produced.

    (BrowserUk:) It could work and would be useful. It would be a departure from the norm of converting numerics to and from their binary representation.

    There are some issues as to what should happen if you specified pack 'Y4', 100000; or pack 'Y4', 32.0 but actually that perhaps suggests a way around the lack of remaining letters that also has an existing precedent.

    To skip a complex structure--say consisting of 2 shorts and float--the syntax is X[vvf], meaning skip enough bytes to cover 2 shorts and a float. Ie. 2+2+4 = 8 bytes.

    To get your hexified numeric, the syntax could be pack 'H[v]', 32, meaning treat the number as a a 16-bit int and hexify it; thereby producing your 4 bytes of output.

    The nice thing is that this then extends naturally to pack 'H[V]', 32; to produce 8-bytes of hex. And pack 'H[Q]', 123456789012345; and even pack 'H[f]', 1234.56e78; and so on.

    And once that is accepted, this further extends to the other bane of pack/unpack; binary. With 'B[v] b[V] B[d]' etc. And a quick peruse of the docs suggest] that 'O' isn't currently used, so maybe 'O[v] O[Q]' might be useful also.

    Now all you've got to do is: knock up the patch; get it by p5p; and wait for it to make it into a build :)

    (End of thread summary).

    Perhaps, rather than having to reproduce much of the complexity of the [v] [Q] etc. notation, the format item could "functionally compose" with the item that followed. (Not so different from BrowserUk's Re^3 suggestion, but all carried out within pack/unpack.) So, whatever string the following pack format item produces gets turned into printable hex (or octal or binary, I suppose), and unpack would turn the printable string into the (usually unprintable string) that the next item expects to unpack. I haven't actually looked at the pack/unpack code, but this has the "feel" of something that should be able to piggy-back off existing code to do all the "hard work".

[reply]
[d/l]

      You've done no one any favours but "popping a few levels". This is a hierarchal forum and, used correctly, all posts are "visible" regardless of how deep they get,

      As for the idea of functional composition of pack formats. You'd need to post a couple of examples of your intent before I would be able to interpret your meaning?

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Here's what I have in mind. pack, in the abstract, takes a substring of the format string, starting at some offset, and an argument to pack, converts the argument to a string that gets appended to the result returned by pack, and updates the offset in the format string so the next argument can be processed. Maybe that's also how the code works, maybe not, as I said, I haven't yet looked. If it were possible for a format item, let's call it Y, to "intercept" the string produced by whatever what follows it in the format string, and return the "printable hex" version as the result (pretty much as the H format does now to the string argument it expects in the argument list), then we would already have all the integer width/endian-ness/sign-edness processing done for us by existing code. Maybe Y[o] could get us octal or Y[X] would yield hex with uppercase A-F. But the heavy lifting would be done by whatever follows in the format string. So 

          pack("Yv", -1);      # yields "ffff"
  pack("YN", -1);      # yields "ffffffff"
  pack("Y[X]v", -1);   # maybe yields "FFFF"
  pack("Y[o]v", -1);   # maybe yields "177777"
  pack("Y[b]c", 15);   # maybe yields "00001111"

        [download]

        The point being, what follows the Y construct determines the content (including length) of the string, and the Y construct turns that string, wherever it came from, into something printable.

        Unpack would do the reverse, but it would be a little more complicated, because it would have to determine, from what follows in the format string, how long a string is expected. Knowing that, it could convert a printable string into whatever string the following format expects.

        This would make the job of the Y format item nearly trivial, but it would put some demands on the pack/unpack code, like being able to determine the length of the string a format string expects to unpack. If we restricted what could follow Y to a single letter (no skipping, repetitions, etc.), that could easily be provided, and would be adequate for handling all the integer and floating point values, which are the major suspects in packing into unprintable strings.

        BrowserUk's suggestions about how to generalize the concept are the sort of wisdom I was hoping for from this forum. I'd like to refine the idea some before digging into the code and maybe bouncing it off p5p.

[reply]
[d/l]

In Section Seekers of Perl Wisdom