Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

When I use this regex to substitue a micro symbol for the letter u, it works but leaves a unicode special character. How can I remove that.
$contents =~ s/(\x{B5}|\x{BC})/u/g;
μ �

Replies are listed 'Best First'.
Re: Substituting unicode character leaves special block
by haj (Vicar) on Mar 17, 2023 at 18:37 UTC

    I guess that your original text is UTF-8 encoded and you should decode it to Perl characters before performing regex substitutions. The character μ happens to be one of those where the UTF-8 encoding 0xc2 0xb5 contains the ISO-8859-1 encoding 0xb5. So, if you run s/(\x{B5}|\x{BC})/u/g; on UTF-8 encoded µ, then Perl replaces the 0xb5 as you asked for, leaving the 0xc2. This single character 0xc2 is not valid UTF-8, if you print it on a terminal which expects UTF-8 then you get the unicode replacement character.

    use 5.020; use Encode; sub show { my ($char) = @_; printf "%s = %vx\n",encode('UTF-8',$char),$_[0]; } my $emu = "\xc2\xb5"; show $emu; # µ = c2.b5 my $mu = decode('UTF-8',$emu); show $mu; # µ = b5 $emu =~ s/(\x{B5}|\x{BC})/u/g; # Don't do this! show $emu; # Âu = c2.75 $mu =~ s/(\x{B5}|\x{BC})/u/g; # That's how to do it show $mu; # u = 75
Re: Substituting unicode character leaves special block
by kcott (Archbishop) on Mar 18, 2023 at 00:54 UTC

    Possibly a typo in either your code or when posting; regardless, \x{BC} is 1/4 as a single character, not mu.

    Both substitution and transliteration work, as shown in the code below. [I used the /r option for demonstration purposes. You may not need or want this; but, if you do, you'll need Perl v5.14 or later to use it (see "perl5140delta: Non-destructive substitution").]

    $ perl -E ' use strict; use warnings; use utf8; use open OUT => qw{:encoding(UTF-8) :std}; use Unicode::UCD "charinfo"; say "\x{B5} = ", charinfo(0xB5)->{name}; say "\x{BC} = ", charinfo(0xBC)->{name}; say "\x{3BC} = ", charinfo(0x3BC)->{name}; my $contents = "\x{B5} \x{BC} \x{3BC}"; print "Original \$contents: "; say $contents; print "Non-desctructive s///: "; say $contents =~ s/[\x{B5}\x{3BC}]/u/gr; print "Unchanged \$contents: "; say $contents; print "Non-desctructive y///: "; say $contents =~ y/\x{B5}\x{3BC}/u/r; print "Unchanged \$contents: "; say $contents; '

    Output:

    µ = MICRO SIGN
    ¼ = VULGAR FRACTION ONE QUARTER
    μ = GREEK SMALL LETTER MU
    Original $contents:    µ ¼ μ
    Non-desctructive s///: u ¼ u
    Unchanged $contents:   µ ¼ μ
    Non-desctructive y///: u ¼ u
    Unchanged $contents:   µ ¼ μ
    

    See also the Unicode PDF code charts: "C1 Controls and Latin-1 Supplement" (for x{B5} & \x{BC}); and, "Greek and Coptic" (for \x{3BC}).

    — Ken

Re: Substituting unicode character leaves special block
by LanX (Saint) on Mar 17, 2023 at 17:39 UTC
    If $contents has the UTF-8 flag activated - i.e. it was properly decoded - then replace the two characters as such.

    Please shows us the output of Devel::Peek::Dump() before and after the substitution.

    See https://perldoc.perl.org/utf8 for more on the two flavors of strings in Perl, octet strings and character strings.

    Cheers Rolf
    (addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
    Wikisyntax for the Monastery

Re: Substituting unicode character leaves special block
by NetWallah (Canon) on Mar 17, 2023 at 18:34 UTC
    This works for me:
    perl -E '(my $mu="μ")=~tr/\x{B5}\x{BC}\x{ce}/uu/d; say unpack "H* +",$mu;say $mu' 75 u
    Note: the code tags seem to mangle my initialization of $mu to"μ".
    I added deletion of 0XCE - perhaps a Unicode-enlightened monk can explain how that stays behind.

                    "These opinions are my own, though for a small fee they be yours too."

Re: Substituting unicode character leaves special block
by hippo (Archbishop) on Mar 17, 2023 at 19:41 UTC
    use strict;
    use warnings;
    
    use Test::More tests => 1;
    
    my $micro = 'μ';
    
    $micro =~ s/μ/u/;
    
    is $micro, 'u';
    

    🦛

      You might want to clarify the encoding of your example. You do not use utf8;, yet you have a literal 'µ'. So, if you have an editor which saves this as UTF-8, then $micro is a string of two characters, and your substitution replaces the very same two characters with an u. This is what I'd call "works by accident".

      To make it more interesting, your code contains the HTML escape &956; which is GREEK SMALL LETTER MU and not the MICRO SIGN, &b2; from the question. These look pretty much the same, but are different characters. Your GREEK SMALL LETTER MU can not be represented in Perl's default character set!

        You might want to clarify the encoding of your example.

        A fair point: it is utf-8. But I don't think that matters overly because ...

        You do not use utf8;, yet you have a literal 'µ'. So, if you have an editor which saves this as UTF-8, then $micro is a string of two characters, and your substitution replaces the very same two characters with an u. This is what I'd call "works by accident".

        If it works by accident then it does so deliberately :-) What I have tried to demonstrate here is that you can substitute a given symbol in code simply with a regex. But of course this depends entirely on where your string/corpus comes from, how it's encoded (or not) and how the pattern is constructed. Our anonymous OP has only provided the last of these, so it becomes guesswork in regards to what they are really doing.

        And you are right about the difference between the mu and micro characters in unicode. I rarely type the micro symbol - much more frequently I write it by hand where I use a mu and always have done. The difference (to me, anyway) is a semantic one rather than a typographical one. I probably should have used µ in the example instead of μ - mea culpa.


        🦛

        I agree that use utf8; would be better.

        > which saves this as UTF-8, then $micro is a string of two characters,

        don't you mean one character encoded with 2 bytes? °

        > To make it more interesting

        the most interesting thing is that he wisely used pre-tags and not code-tags, because code-sections here fail with utf-8 characters ... :/

        update

        °) OK, you mean when Perl loads this file, it will interpret the 2 bytes as 2 random characters, because use utf8 is missing. Yes!

        Cheers Rolf
        (addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
        Wikisyntax for the Monastery

Re: Substituting unicode character leaves special block
by Anonymous Monk on Mar 17, 2023 at 18:03 UTC
    Fixed by adding +
    $contents =~ s/(\x{B5}|\x{BC})+/u/g;