Re: UTF-8 text files with Byte Order Mark

A BOM is part of the text and it's a (sort of) valid character "ZERO WIDTH NON-BREAKING SPACE". Your best bet is just to strip it off since it's use (aside from providing a BOM) isn't recommended anyway:

while (my $line = <>) {
   $line =~ /^\x{FEFF}//; # strip BOM
   # rest
}
[download]

"What should it profit a man, if he should win a flame war, yet lose his cool?"

Comment on Re: UTF-8 text files with Byte Order Mark Download Code

Replies are listed 'Best First'.
Re^2: UTF-8 text files with Byte Order Mark by muba (Priest) on Feb 13, 2007 at 20:21 UTC
Yeah, this works, except that the BOM indeed is a three-bytes thing as said above. So the code, that seems to work, now looks like this: `while (my $line = <$rulesFH>) { if ($. == 1) { # Remove Byte Order Mark if it's there use Encode; my $octets = encode("utf8", $line); $octets =~ s/^\x{ef}\x{bb}\x{bf}//; $line = decode("utf8", $octets); } # rest... }` [download]	[reply] [d/l]
Re^3: UTF-8 text files with Byte Order Mark by ikegami (Patriarch) on Feb 13, 2007 at 20:48 UTC
`my $octets = encode("utf8", $line); $octets =~ s/^\x{ef}\x{bb}\x{bf}//; $line = decode("utf8", $octets);` [download] is the same thing as `my $BOM = decode("utf8", "\x{ef}\x{bb}\x{bf}"); $line =~ s/^$BOM//;` [download] is the same thing as `my $BOM = chr(0xFEFF); $line =~ s/^$BOM//;` [download] is the same thing as `$line =~ s/^\x{FEFF}//;` [download] which is what I gave you. Much simpler!	[reply] [d/l] [select]
Re^4: UTF-8 text files with Byte Order Mark by muba (Priest) on Feb 13, 2007 at 21:01 UTC
Meh. Indeed, I didn't realise that. Thank you!	[reply]
Re^4: UTF-8 text files with Byte Order Mark by Anonymous Monk on Sep 30, 2011 at 18:30 UTC
Thank you!!! This saved me a lot of trouble. I am also trying to strip out these UTF-8 byte order mark characters, which google docs puts in by default to downloaded text files. By the way I found that \x{FEFF} was not the same as \x{ef}\x{bb}\x{bf}	[reply]
Re^5: UTF-8 text files with Byte Order Mark by ikegami (Patriarch) on Oct 01, 2011 at 21:53 UTC
A reply falls below the community's threshold of quality. You may see it by logging in.