in reply to Re^2: Malformed UTF-8
in thread Malformed UTF-8

Maybe your perl interpreter is buggy. Can you try running the code I posted above and see if that works?

Replies are listed 'Best First'.
Re^4: Malformed UTF-8
by spiros (Beadle) on May 15, 2007 at 17:15 UTC
    bunny:/tmp spiros$ perl u.pl SV = PV(0x1801460) at 0x180bcf0 REFCNT = 1 FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8) PV = 0x30f160 "ba\303\261o"\0 [UTF8 "ba\x{f1}o"] CUR = 5 LEN = 6 match
    What does the utf8 upgrade function do?
      utf8::upgrade just forces the string to be utf-8 (otherwise it'll probably latin-1 - i.e. 1-byte encoding) in an attempt to match your problem. Since it doesn't reproduce the warning/error you're reporting I must have missed something.

      I note there are 2 variables in your code. Are you SURE the problem arises when both variables are "baņo" AND are both utf-8 encoded? In other words, check the Devel::Peek::Dump output from both variables.

        This is the output for both $term and $token just before the error:
        TOKEN: SV = PVMG(0x1c6cca0) at 0x1ab6a28 REFCNT = 1 FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8) IV = 0 NV = 0 PV = 0x2dd2f80 "ba\303\261o"\0 [UTF8 "ba\x{f1}o"] CUR = 5 LEN = 15 MAGIC = 0x2dc95f0 MG_VIRTUAL = &PL_vtbl_utf8 MG_TYPE = PERL_MAGIC_utf8(w) MG_LEN = 4 -------------------- TERM: SV = PVIV(0x18b8e20) at 0x1ab69c8 REFCNT = 1 FLAGS = (PADBUSY,PADMY,POK,pPOK) IV = 0 PV = 0x2dd3110 "ba\303\261o"\0 CUR = 5 LEN = 12 --------------------
        It appears $term is not actually UTF-8 encoded when this occurs. Additionally, is it me or does
        [UTF8 "ba\x{f1}o"]
        look wrong ? From what i recall, the UTF8 part of Dump should show the actual word, meaning bano (accented n) and not the encoding.