spiros has asked for the wisdom of the Perl Monks concerning the following question:

Oh ye who are wise and enlightened, I am doing several regex matches using Spanish text as read from a configuration file and keep getting this message: 
Malformed UTF-8 character (unexpected non-continuation byte 0x00, imme
+diately after start byte 0xc3) in pattern match (m//) at /Users/spiro
+s/.... line 90, <$fh> line 5.

[download]
The actual code that does the regex is : 
return $n if ( $token =~ m/^$term/i );

[download]
I took some extra steps and in order to make sure all the strings I used where in UTF-8, i used Devel::Peek. Indeed all the SV's have the UTF-8 flag on: 
SV = PVMG(0x1c6c2a0) at 0x1ab6a28
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8)
  IV = 0
  NV = 0
  PV = 0x2dd2d80 "alto"\0 [UTF8 "alto"]
  CUR = 4
  LEN = 15
  MAGIC = 0x2dd30b0
    MG_VIRTUAL = &PL_vtbl_utf8
    MG_TYPE = PERL_MAGIC_utf8(w)
    MG_LEN = 4

[download]
etc. Right before the warning, this was the output from Devel::Peek for that particular SV: 
SV = PVMG(0x1c6c2a0) at 0x1ab6a28
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8)
  IV = 0
  NV = 0
  PV = 0x2dd2d80 "ba\303\261o"\0 [UTF8 "ba\x{f1}o"]
  CUR = 5
  LEN = 15
  MAGIC = 0x2dd30b0
    MG_VIRTUAL = &PL_vtbl_utf8
    MG_TYPE = PERL_MAGIC_utf8(w)
    MG_LEN = 4

[download]
Some reading here and there revealed that this happens when a string contains 'wide characters' as in, character sequences thus regex wont work. I am truly, madly, deeply stuck here and this is making me ANGRY and SCARED. Your help would be much appreciated.

Replies are listed 'Best First'.
Re: Malformed UTF-8
by Joost (Canon) on May 15, 2007 at 17:04 UTC
    So what's in $term and which version of perl are you using? I can't reproduce your problem: 

    use strict;
use warnings;
use Devel::Peek 'Dump';
my $token = "ba\x{f1}o";
utf8::upgrade($token);

Dump($token);
my $term = $token; # note: i have no clue what's in $term
print "match\n" if $token =~ /^$term/i;

    [download]
    update: output from that: 
    SV = PV(0x814d988) at 0x814d524
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,POK,pPOK,UTF8)
  PV = 0x8160848 "ba\303\261o"\0 [UTF8 "ba\x{f1}o"]
  CUR = 5
  LEN = 6
match

    [download]
    update2: changed code so that $token and $term are the same. does not influence the outcome at all.

    "What should it profit a man, if he should win a flame war, yet lose his cool?"
[reply]
[d/l]
[select]
      Thank you for your reply. This is perl -v : 
      This is perl, v5.8.6 built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)

      [download]
      (let me know if you need a -V output) $term contains 'baņo'. This is the Dump for $term before the error: 
      SV = PVIV(0x18b8e20) at 0x1ab69c8
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,POK,pPOK)
  IV = 0
  PV = 0x2dd2ef0 "ba\303\261o"\0
  CUR = 5
  LEN = 12

      [download]
      Actually, this is weird, I just realized, this error occurs when they match since both $token and $term are identical (or should be at least). Hm.
[reply]
[d/l]
[select]
[reply]

Back to Seekers of Perl Wisdom