Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Modern best practices for multilingual regexp alphabetical character matching?

by ikegami (Patriarch)
on Jan 12, 2009 at 21:40 UTC ( #735814=note: print w/replies, xml ) Need Help??


in reply to Modern best practices for multilingual regexp alphabetical character matching?

The data I'm working with will all be UTF-8, if that makes a difference.

Make sure it's decoded using one or more of the following

use utf8; # Treat the source code as UTF-8 use open ':std', ':locale'; # Treat STD* as per locale use open ':std', ':encoding(UTF-8)'; # Treat STD* as UTF-8 use open IO => ':encoding(UTF-8)'; # Treat files as UTF-8 by default open(my $fh, '<:encoding(UTF-8)', $qfn) # Treat a file as UTF-8 utf8::decode(my $text = $encoded_text) # Treat a string as UTF-8 or die;

And make sure the string us stored internally as UTF-8.

utf8::upgrade($s); # Use UNICODE semantics

(No need to do use utf8; to use utf8:: functions. use utf8; means the source is in UTF-8.)

If you do those two things, regexp will use UNICODE semantics, so \w and character classes will match accented letters, etc.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://735814]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (5)
As of 2023-06-10 23:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How often do you go to conferences?






    Results (39 votes). Check out past polls.

    Notices?