Isn't the unpack 'A*' redundant ? (But wouldn't 'a*' be better ?)

I can see the logic that what pack produces should really be unpacked before being used. Indeed, it occurred to me that unpack 'a*',... might do something bright with UTF-8. Which set me on a small quest to discover how to convert UTF-8 in hex characters to utf8 characters....

---

The following:

  use strict ;
  use warnings ;

  use Encode qw(_utf8_on) ;

  for my $r ("\xC2\xAB \x61\x68\x61 \xC2\xBB", "\xC2\x7E \x61\x68\x61 
+\x80\xC0") {
    for my $utf (0..1) {
      _utf8_on($r) if $utf ;
      printf "'%s', %d/%d %s\n", raw(unpack('a*', $r)) ;
    } ;
  } ;

  sub raw {
    my ($s) = @_ ;

    my ($b, $q) ;
    { use bytes ;
      $b = length($s) ;
      $q = join '', map { ($_ >= 0x20) && ($_ <= 0x7E) ? chr($_) : spr
+intf('\\x%02X', $_)
                        } unpack('C*', $s) ;
    } ;

    return ($q, length($s), $b, utf8::is_utf8($s) ? 'utf8' : 'not utf8
+') ;
  } ;
[download]

gives:

  '\xC2\xAB aha \xC2\xBB', 9/9 not utf8
  '\xC2\xAB aha \xC2\xBB', 7/9 utf8
  '\xC2~ aha \x80\xC0', 9/9 not utf8
  Malformed UTF-8 string in unpack at ...

showing that if the string being unpacked is utf8, the result is utf8 (or error, if not valid utf8).

I found, however, that pack 'H*',... returns a byte (not utf8) string, no matter what the input(s). This seems, on the whole, reasonable.

I tried a number of things to try to get unpack('a*', pack('H*', $foo)) to return utf8, ...

  my $s = "C2AB2061686120C2BB" ; _utf8_on($s) ;

  for my $unp ('a*', 'U0a*', 'C0a*') {
    my ($q, $b, $l, $u) = raw(unpack($unp, pack('H*', $s))) ;
  print "unpack('$unp', pack('H*', \$s)) -> '$q', $l/$b $u\n" ;
} ;
[download]

but to no avail:

  unpack('a*', pack('H*', $s)) -> '\xC2\xAB aha \xC2\xBB', 9/9 not utf8
  unpack('U0a*', pack('H*', $s)) -> '\xC3\x82\xC2\xAB aha \xC3\x82\xC2\xBB', 13/13 not utf8
  unpack('C0a*', pack('H*', $s)) -> '\xC2\xAB aha \xC2\xBB', 9/9 not utf8

but note that unpack 'U0a*' is "upgrading" (as in utf8::upgrade()) the bytes to UTF-8.

I found that the trick is to tell pack to return utf8, thus:

  my $s = "C2AB2061686120C2BB" ;

  for my $unp ('a*', 'U0a*', 'C0a*') {
    printf "unpack('$unp', pack('U0H*', $s)) -> '%s', %d/%d %s\n",
        raw(unpack( $unp,  pack('U0H*', $s))) ;
  } ;
[download]

giving:

  unpack('a*', pack('U0H*', C2AB2061686120C2BB)) -> '\xC2\xAB aha \xC2\xBB', 7/9 utf8
  unpack('U0a*', pack('U0H*', C2AB2061686120C2BB)) -> '\xC2\xAB aha \xC2\xBB', 9/9 not utf8
  unpack('C0a*', pack('U0H*', C2AB2061686120C2BB)) -> '\xC2\xAB aha \xC2\xBB', 7/9 utf8

noting that unpack 'U0a*' is treating its input as bytes.

The unpack is still optional, though invalid UTF-8 is treated differently if it's left out, thus:

  for my $s ("C2AB2041686120C2BB", "C27E204168612080C0") {
    printf "pack('U0H*', $s) -> '%s', %d/%d %s\n",
        raw(pack('U0H*', $s)) ;
    printf "unpack('a*', pack('U0H*', $s)) -> '%s', %d/%d %s\n",
        raw(unpack('a*', pack('U0H*', $s))) ;
  } ;
[download]

gives:

  pack('U0H*', C2AB2041686120C2BB) -> '\xC2\xAB Aha \xC2\xBB', 7/9 utf8
  unpack('a*', pack('U0H*', C2AB2041686120C2BB)) -> '\xC2\xAB Aha \xC2\xBB', 7/9 utf8
  Malformed UTF-8 character (unexpected end of string) in length at ../hex-utf.pl line 23.
  pack('U0H*', C27E204168612080C0) -> '\xC2~ Aha \x80\xC0', 7/9 utf8
  Malformed UTF-8 string in unpack at ../hex-utf.pl line 48.

so pack is not checking for valid UTF-8, leaving it as a puzzle for others -- and in this case length() is throwing a warning. On the other hand, unpack is deeply unhappy about invalid UTF-8, and throws an error.

None of this was entirely obvious to me. Hopefully somebody can benefit from my little quest.

---

Returning to the topic of the OP, if I wanted to decode the hex as UTF-8, I think what I would do is:

  sub dehex {
    my ($s) = @_ ;
    $s =~ s/0[xX]((?:[0-9A-Fa-f]{2})+)/pack('U0H*', $1)/eg ;
    return $s if utf8::valid($s) ;
    ... worry ...  return undef ??
  } ;
[download]