length() and Unicode

Lu. has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks, I came across what seems to me a strange behaviour of length().
I am using a simple script to parse an XML file looking like this :

<file ID="A01">
<p>
<s n="0001"> <w POS="a">&#22823;</w> <w POS="n">&#22681;</w> <w POS="f
+">&#20869;&#22806;</w> <c POS="w">&#65293;&#65293;</c> <w POS="ns">&#
+21271;&#20140;&#24066;</w> <w POS="n">&#30417;&#29425;</w> <w POS="n"
+>&#32426;&#23454;</w> <c POS="w">&#65288;</c> <w POS="m">&#19977;</w>
+ <c POS="w">&#65289;</c> </s>
</p>
<p>
<s n="0002"> <w POS="nr">&#30000;</w> <w POS="nr">&#29645;&#39062;</w>
+ </s>
[download]

(the &#thingies are actually chinese characters in the file)

This file is a segmented text with the grammatical function of each word. What I aim to do is print to a file the list of words, with their position and grammatical function. So, the output will look like this :

1    &#22823;    a
2    &#22681;    n
3    &#20869;&#22806;    f
5    &#65293;&#65293;    w
7    &#21271;&#20140;&#24066;    ns
10    &#30417;&#29425;    n
12    &#32426;&#23454;    n
13    &#65288;    w
14    &#19977;    m
15    &#65289;    w
17    &#30000;    nr
18    &#29645;&#39062;    nr
[download]

(the new line is counted as one position, thus resulting in the position 17 for the eleventh item)

I have written the following code, which works A-Ok, except for the position.

#!/usr/bin/perl -w

use encoding 'utf-8';

die "Usage : $0 source_file(XML)\n" unless (@ARGV>=1);

open (FILEIN, "$ARGV[0]") || die "Unable to open source file $ARGV[0] 
+: $!\n";
open (FILEOUT, ">:utf8", "$ARGV[0]_tok_cat.lst") || die "Unable to ope
+n destination file $ARGV[0]_tok_cat.lst : $!";

$pos = 0;
while (<FILEIN>) {
    if (m/^\<s/) {
    $pos++;
    #@tmp = split;
    foreach $item (split) {
        if ($item =~ m/\>.+\</) {
        $cat = $item;
        $item =~ s/POS=\"[a-z]+\">(.+)\<\/[cw]>/$1/;
        $cat =~ s/POS=\"([a-z]+)\">.+\<\/[cw]>/$1/;
        print FILEOUT "$pos\t$item\t$cat\n";
        $pos+=length($item);
        }
    }
    }
}
close FILEIN;
close FILEOUT;
[download]

The input file is encoded in utf-8 and length($string) returns thrice the size of the string it measures. perldoc states it should return the number of characters in the string.

Can anybody explain why length() counts in bytes instead of characters? Is this a bug or did I miss something? I could solve the issue by dividing the returned value by 3, but it feels like It would be cheating...

I use perl version 5.8.7 in Linux, and my locale is fr_FR.UTF-8, if it is of any use.

Comment on length() and Unicode Select or Download Code

Replies are listed 'Best First'.
Re: length() and Unicode by Thelonius (Priest) on Aug 08, 2006 at 17:21 UTC
I think you should open FILEIN with :utf8 also, that is `open (FILEIN, "<:utf8", $ARGV[0]) \|\| die "Unable to open source file $ARGV[0] : $!\n";` [download]	[reply] [d/l]
Re: length() and Unicode by ikegami (Patriarch) on Aug 08, 2006 at 16:37 UTC
It sounds like the input is being treated as a string of bytes (as opposed to a string of characters). Use Encode to convert it to characters. (`_utf8_on` might to the trick if the data is in UTF-8.) ( My statement was accidently negated. Fixed. ) ( Thelonius's solution is better. )	[reply] [d/l]