Hello AnishaM,

A quick search of turns up the module String::BOM which has a strip_bom_from_string function:

use strict;
use warnings;
use utf8;
use String::BOM qw( strip_bom_from_string );

my $string = 'ÿþThe quick brown fox jumped over the unfortunate dog.';
print ">$string<\n";
$string = strip_bom_from_string($string);
print ">$string<\n";
[download]

Output:

22:35 >perl 1697_SoPW.pl
>ÿþThe quick brown fox jumped over the unfortunate dog.<
>The quick brown fox jumped over the unfortunate dog.<

22:35 >
[download]

Note that this will remove the BOM from the beginning of the string only — which is the only place a BOM is supposed to occur within a text stream (see Byte order mark).

Hope that helps,

Since the BOM is unicode, you have to use unicode to remove the BOM:

s/\x{feff}//;
[download]

But if you use vim, removing the BOM is as simple as loading the file, typing

:set nobomb
[download]

and writing the file back to disk.

use utf8;
                                                           
my $string = 'AÿþBÿþÿþC';
$string =~ s/ÿþ//g;
print $string;
[download]

ABC
[download]

I tried this: s/^\N{ZERO WIDTH NO-BREAK SPACE}//;

Works for me (unanchored):

#!/usr/bin/env perl 
use strict;
use warnings;
use Test::More tests => 2;

my $src = "foo\x{feff}bar";

my $res = $src;
$res =~ s/\N{ZERO WIDTH NO-BREAK SPACE}//;
isnt ($res, $src, 'Output differs from src');
is ($res, 'foobar', 'BOM stripped');
[download]

You can always add the anchor in if required (as the BOM should only occur at the start as Athanasius explained).

Hi AnishaM,

Personally I would try to diagnose the issue a little further, since even a simple print of the variable can give misleading results when it comes to diagnosing encoding problems.

If the data is coming from a file, I suggest you try the commands hexdump -C FILENAME or od -Ax -tx1z FILENAME and show us the first few lines.

If the data is coming from some other source and you just have a Perl string, try the code use Devel::Peek; Dump( $string ); and show us that output.

Regards,
-- Hauke D

G'day AnishaM,

TMTOWTDI:

$ perl -Mutf8 -E 'say join "", split /ÿþ/, q{AÿþBÿþÿþC}'
ABC
[download]

See also: Benchmark.

This is an example from some code I wrote to read in a MS file that had a superfluous BOM.

my @file;

{
  my $lh;

  open($lh, "<:utf8:crlf", $LogFn) || do {
    Pe "\nlogfile <%s> not found", $LogFn;
    help;
  };

  @file = grep {
                  s/([^\r]+)$/$1/;
                  m{^\s*$} ? undef : $_;
                } <$lh>;
  close $lh;
}

$file[0] =~ s/^\N{U+FEFF}//;    # UTF-8 BOM
[download]

The stuff immediately after the open was to remove the carriage returns, so I could have normal unix line endings.

"Pe", BTW is part of "P". From what I understand, in newer perls, a new keyword, "err" is weaker version of the same (doesn't allow a format statement).

Hope this is of use.