Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

So, I'm trying (for whatever reason) to split a string into its component characters. This is what I do:

my @letters = split //, @text

[download]

But there are some non-ASCII characters (quite a variety, actually) in the text, so in the code, there are the following lines:

use locale;
use POSIX qw(locale_h);
use POSIX;

use Encode;

my $old_lcall = setlocale(LC_ALL);
my $old_ctype = setlocale(LC_CTYPE);
setlocale(LC_ALL,"cs_CZ.utf8");
setlocale(LC_CTYPE,"cs_CZ.utf8");

# ...the $text variable has also been encoded as UTF-8 (the 
# hyphenated version), using the encode function provided 
# by use Encode;

my @letters = split //, $text;

# and at the end of the code, set the locales back:
setlocale(LC_ALL, $old_lcall);
setlocale(LC_CTYPE, $old_ctype);

[download]

Looking at the output, split reads the string in $text byte-by-byte (while sort, for example, works as it should under the locale). So the question is, o wise Perl monks, how do I get split to obey the locale as well?

Replies are listed 'Best First'.
Re: Split is ignoring locale settings
by Anonymous Monk on Oct 27, 2011 at 17:31 UTC

    Forget about locales. They are from an era before Unicode. Use the other modern tools Perl and CPAN give you.

    That being said, your problem is that you simply do not operate on a Perl string, but a buffer of bytes. When you split that, you get bytes. However, you want to split a string into characters.

    The difference is in the need of decoding your input from bytes to characters. Learn about the whole topic of encoding at p3rl.org/UNI.

    Sample solution: 

        use Encode qw(decode);

    my $bytes = "Chrt pln skvrn vtrhl skrz trs chrp v \xC4\x8Dtvr\xC5\xA5 Kr\xC4\x8D."
    my $text  = decode 'UTF-8', $bytes;

    # alternatively:
    # use utf8;
    # my $text = 'Chrt pln skvrn vtrhl skrz trs chrp v čtvrť Krč.';

    my @characters = split //, $text;
    # (
    #    'C', 'h', 'r', 't', ' ', 'p', 'l', 'n', ' ', 's', 'k', 'v', 'r', 'n',
    #    ' ', 'v', 't', 'r', 'h', 'l', ' ', 's', 'k', 'r', 'z', ' ', 't', 'r',
    #    's', ' ', 'c', 'h', 'r', 'p', ' ', 'v', ' ', "\x{10d}", 't', 'v', 'r',
    #    "\x{165}", ' ', 'K', 'r', "\x{10d}", '.'
    # )

    Since everything is UTF-8, it's a bit difficult for you as a beginner to grok what's going on. Exchange the following two lines, perhaps it will enlighten you quickly. 

        my $bytes = "Chrt pln skvrn vtrhl skrz trs chrp v \xe8tvr\xbb Kr\xe8.";
    my $text  = decode 'Latin-2', $bytes;
[reply]
Re: Split is ignoring locale settings
by Anonymous Monk on Oct 27, 2011 at 17:16 UTC

    Edit: problem solved. Use:

    my @letters = split //, decode("UTF-8",$text);

    [download]

[reply]
[d/l]

Back to Seekers of Perl Wisdom