After hours of struggling with a seemingly simple problem involving utf8 I finally made it work -- but I have no clue why. Allow me to explain.

I have a script which scrapes a particular web site for data about communication links. The data is utf-8 encoded and contains (among a great many other things) the unicode character 'GREEK SMALL LETTER MU' (U+03BC) which (after being scraped, put through several scripts, stored in MySQL, later extracted and presented on a web page) renders as "μ". My co-workers didn't really mind but after a couple of years it started to annoy me so much I reached the point where I just had to fix it. Today was that day.

I wanted to replace all occurences of this character with either unicode character 'MICRO SIGN' (U+00B5) which renders as expected, or (even better) simply with the HTML entity µ.

The method in question produces clickable links to present each commlink in many different contexts.

sub commlink {
  my $self = shift;
  return "" unless $self->{'id'};
  my $label = $self->{'label'};
  $label =~ s/\x{00b5}/\µ/g;
  $label =~ s/\x{03bc}/\μ/g; # Looks almost exactly the same as &mi
+cro;
  return "<A href=\"commlink.html?id=".$self->{'id'}."\" class=\"".$se
+lf->{'state'}."\">".$label."</A>";
}

[download]

I knew the data stored in MySQL was utf8, the string was untoched and the web page charset was specified as utf8. If I tried to change it, the norwegian characters on the same page would become garbled so I knew the encoding setting was sent and detected properly.

So... utf8 in, no encoding/decoding or string mangling prior to the regex... and still the regex didn't match.

The solution?

sub commlink {
  my $self = shift;
  return "" unless $self->{'id'};
  my $label = $self->{'label'};
  $label = decode('utf8', $label); # Why? It's already utf8 and I need
+ it to stay utf8
  $label =~ s/\x{00b5}/\&micro;/g;
  $label =~ s/\x{03bc}/\&mu;/g; # Looks almost exactly the same as &mi
+cro;
  return "<A href=\"commlink.html?id=".$self->{'id'}."\" class=\"".$se
+lf->{'state'}."\">".$label."</A>";
}

[download]

My question is... Why?! Before decoding the utf8 string, how could the string go from input to output unchanged but fail to match the regex? Why do I need to decode the utf8 string to match an utf8 character when the string already prints as an utf8 character? This is so confusing...

UPDATE:

OK, thanks for the pointers. It sounds so very very simple in theory, but in practice... This system is made up from more than 50 different scripts and modules that shuffle data back and forth and present it via HTML, SVG, generates javascript, text messages, emails and what have you. After I started trying to fix things to "do it right" then absolutely everything broke. I'm going to need weeks to get on top of this.

This is exactly why I have always hated Unicode. Why, oh why could I not have left this stupid bug alone.

-- FloydATC

Time flies when you don't know what you're doing

In reply to Matching/replacing a unicode character only works after decode() by FloydATC

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.