<file ID="A01">
<p>
<s n="0001"> <w POS="a">&#22823;</w> <w POS="n">&#22681;</w> <w POS="f
+">&#20869;&#22806;</w> <c POS="w">&#65293;&#65293;</c> <w POS="ns">&#
+21271;&#20140;&#24066;</w> <w POS="n">&#30417;&#29425;</w> <w POS="n"
+>&#32426;&#23454;</w> <c POS="w">&#65288;</c> <w POS="m">&#19977;</w>
+ <c POS="w">&#65289;</c> </s>
</p>
<p>
<s n="0002"> <w POS="nr">&#30000;</w> <w POS="nr">&#29645;&#39062;</w>
+ </s>
[download]

This file is a segmented text with the grammatical function of each word. What I aim to do is print to a file the list of words, with their position and grammatical function. So, the output will look like this :

1    大    a
2    墙    n
3    内外    f
5    －－    w
7    北京市    ns
10    监狱    n
12    纪实    n
13    （    w
14    三    m
15    ）    w
17    田    nr
18    珍颖    nr
[download]

(the new line is counted as one position, thus resulting in the position 17 for the eleventh item)

I have written the following code, which works A-Ok, except for the position.

#!/usr/bin/perl -w

use encoding 'utf-8';

die "Usage : $0 source_file(XML)\n" unless (@ARGV>=1);

open (FILEIN, "$ARGV[0]") || die "Unable to open source file $ARGV[0] 
+: $!\n";
open (FILEOUT, ">:utf8", "$ARGV[0]_tok_cat.lst") || die "Unable to ope
+n destination file $ARGV[0]_tok_cat.lst : $!";

$pos = 0;
while (<FILEIN>) {
    if (m/^\<s/) {
    $pos++;
    #@tmp = split;
    foreach $item (split) {
        if ($item =~ m/\>.+\</) {
        $cat = $item;
        $item =~ s/POS=\"[a-z]+\">(.+)\<\/[cw]>/$1/;
        $cat =~ s/POS=\"([a-z]+)\">.+\<\/[cw]>/$1/;
        print FILEOUT "$pos\t$item\t$cat\n";
        $pos+=length($item);
        }
    }
    }
}
close FILEIN;
close FILEOUT;
[download]

The input file is encoded in utf-8 and length($string) returns thrice the size of the string it measures. perldoc states it should return the number of characters in the string.

Can anybody explain why length() counts in bytes instead of characters? Is this a bug or did I miss something? I could solve the issue by dividing the returned value by 3, but it feels like It would be cheating...

I use perl version 5.8.7 in Linux, and my locale is fr_FR.UTF-8, if it is of any use.