in reply to Re: Matching  & € type characters with a regex
in thread Matching  & € type characters with a regex

Ok. Sooooo... what do I do? I can go though and find all the characters and do s/Â//gsi for each character. Or, is there an easier way to match these types of characters?
  • Comment on Re^2: Matching  & € type characters with a regex

Replies are listed 'Best First'.
Re^3: Matching  & € type characters with a regex
by moritz (Cardinal) on Feb 12, 2009 at 18:43 UTC
    First you have to understand what character encodings are, and how they are handled in Perl.

    I've written this article to explain that, and there's also a lot of other useful information: perluniintro, Encode, perlunicode.

Re^3: Matching  & € type characters with a regex
by ikegami (Patriarch) on Feb 12, 2009 at 18:52 UTC
    If you decode the input as I suggested, you won't have any "Â" or even "®", just the single character those bytes represent. There isn't anything to search and replace.
      This isn't the result of some processing I have done. This is what I have, a bunch of files that fell into my lap that already look like this. I have to clean them up.

        They are already clean if you view them correctly. It's like reading "Comment çà va?" and saying it's gibberish because it doesn't look English. It's not gibberish, it's just not English. What you have isn't junk, it's just not US-ASCII, iso-latin-1, cp1252, Shift_JIS or whatever else you might want it to be.

        Just like "Comment çà va?" can be translated to English, is possible to translate what you have to US-ASCII/​iso-latin-1/​cp1252/​Shift_JIS/​etc. The translation may not be perfect. You may encounter 1:0, 1:N and N:1 relations.

        So the question is: to what encoding would you like it translated to? What follows are a couple of methods for converting your text. The latter allows you to configure (via from_to's fourth argument) what happens when a character in the source can't be represented by the destination encoding. See the docs for details.

        use strict; use warnings; my $src_enc = 'UTF-8'; my $dst_enc = '...'; binmode(STDIN, ":encoding($src_enc)"); binmode(STDOUT, ":encoding($dst_enc)"); print while <STDIN>;

        or

        use strict; use warnings; use Encode qw( from_to ); my $src_enc = 'UTF-8'; my $dst_enc = '...'; while (<>) { from_to($_, $src_enc, $dst_enc); print; }

        Update: Added code. Cleaned up some sentences.