Re: French Accents in perl

Hi,
I´ve found the same trouble working with portuguese acents. I´ve concluded that, in general, usefull resources like \w and \W do not work well for utf8 characters. So, forget them and do the job "mannualy". It happens because Perl 5 has not yet serious UNICODE support, I guess.
May appear strange, but cleaning the strings from \s and \n\r, avoiding using them directly, has helped me. For example:

my reg1= /r\n/;#needed for UTF-8 cleaning
my $reg2 = /s/;#see http://unicode.org/reports/tr13/tr13-5.html

foreach (@words) {
    unless ($_ eq "s+"|$_ eq "0"|$_ eq $reg1|$_ eq $reg2){ #last clean
+ing
    ...
}
[download]

Comment on Re: French Accents in perl Download Code

Replies are listed 'Best First'.
Re^2: French Accents in perl by graff (Chancellor) on Nov 03, 2007 at 04:26 UTC
I'm sorry, but I must disagree. Based on what you've just said, the conclusion I would make is that you have not yet tried seriously to learn how perl 5.8 handles unicode; and you seem to be having trouble with regexes in general, as well. A regex like `/r\n/` (a letter "r" followed by a line-feed) has nothing whatsoever to do with "UTF-8 cleaning". I realize you probably meant `/\r\n/` (CRLF), but that has nothing to do with utf8 cleaning either. (Oh, and there's a syntax error in that line.) I don't know what you're getting at with the `$reg2 = /s/` -- did you forget a backslash there as well? The foreach loop makes no sense at all, and I don't see how any of this (or Mark Davis's very useful essay on "Unicode Newline Guidelines") is at all relevant to accented letters. The "\w" and "\W" work as expected on utf8 strings -- \w matches anything that is a letter or digit, no matter if it's ASCII, extended Latin, Greek, Arabic, Thai or whatever, and \W matches anything that is not a letter or digit. But you have to know for sure that you are applying the regex to a utf8 string that has been flagged internally by Perl as being a utf8 string. If you don't understand what that means, it's not surprising that you've had trouble with this stuff.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: French Accents in perl
by graff (Chancellor) on Nov 03, 2007 at 04:26 UTC

A regex like /r\n/ (a letter "r" followed by a line-feed) has nothing whatsoever to do with "UTF-8 cleaning". I realize you probably meant /\r\n/ (CRLF), but that has nothing to do with utf8 cleaning either. (Oh, and there's a syntax error in that line.)

I don't know what you're getting at with the $reg2 = /s/ -- did you forget a backslash there as well? The foreach loop makes no sense at all, and I don't see how any of this (or Mark Davis's very useful essay on "Unicode Newline Guidelines") is at all relevant to accented letters.

The "\w" and "\W" work as expected on utf8 strings -- \w matches anything that is a letter or digit, no matter if it's ASCII, extended Latin, Greek, Arabic, Thai or whatever, and \W matches anything that is not a letter or digit. But you have to know for sure that you are applying the regex to a utf8 string that has been flagged internally by Perl as being a utf8 string. If you don't understand what that means, it's not surprising that you've had trouble with this stuff.

[reply]
[d/l]
[select]