Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re: length() miscounting UTF8 characters?

by wjw (Priest)
on Apr 27, 2014 at 21:49 UTC ( [id://1084049]=note: print w/replies, xml ) Need Help??


in reply to length() miscounting UTF8 characters?

Having no experience with this, I thought I would explore a bit. So I did the following:
#!/usr/bin/perl use strict; use warnings; use open IO => ':utf8'; while(<DATA>) { chomp; (my $nonenglish = $_) =~ s/[A-Za-z]//g; my @chars = split(//,$nonenglish); my $chars = scalar(@chars); print scalar(@chars), " $nonenglish\n"; } __DATA__ æ æð æða æðaber æðahnútur æðakölkun æðardúnn æðarfugl æðarkolla æðarkóngur æðarvarp æði æðimargur æðisgenginn æðiskast æðislegur æðrast æðri æðrulaus æðruleysi æðruorð æðrutónn æðstur æður æfa __END__

Seems split sees those letters as two chars also, which makes sense now that I think of it... . Guess I have some things to learn about UTF8!
Thanks for the opportunity! Sorry this is not all that helpful. Suppose one could take the character count and just divide by two ...
$chars = $chars / 2; print "$chars $nonenglish\n";
...

Update: Might also take a look at CPAN Test UTF8 and related...

...the majority is always wrong, and always the last to know about it...
Insanity: Doing the same thing over and over again and expecting different results...

Replies are listed 'Best First'.
Re^2: length() miscounting UTF8 characters?
by AppleFritter (Vicar) on Apr 27, 2014 at 22:08 UTC

    Yes, simply dividing by two would work here (and that's what I've been doing, mentally), but that's only because all the non-English characters encountered here are encoded as two bytes in UTF8. As soon as there'd be 3- or 4-byte characters, it'd not work anymore.

    Thanks for your help! I'll take a look at that module.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1084049]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2024-04-18 20:43 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found