Greetings monks,

I'm working with a CSV to XML converter, but I'm having problems when reading UTF-8 files that have an BOM character in the beginning of data.

I'm using the module Text::CSV_XS to read the CSV file and parse the data. I'm expecting the CSV file to be in UTF-8 and dealing with that as expected:

    $csv_file = $cfg->val( 'General', 'inputDir' ) . '\\' . $csv_file;

    my $csv = Text::CSV_XS->new( { binary => 1 } )
      or die "Cannot use CSV: " . Text::CSV_XS->error_diag();

    open( my $fh, '<:utf8', $csv_file )
      or die "Cannot read $csv_file: $!\n";

[download]

As soon as I run the program, I got the error:

# CSV_XS ERROR: 2034 - EIF - Loose unescaped quote @ pos 4

The CSV file has double quotes around the fields, but they are not written inside the double quotes as the error message says.

If I remove the double quotes from the registries, everything goes as expected.

Further testing showed me that removing the BOM of the UTF-8 file and maintaining the double quotes (this is desirable) the program can process the registries without any issue.

While I have an workaround, this seems a bit odd for me. I already had problems with BOM character when dealing with UTF-16 files and the fix for it was opening the file with a:

open( my $fh, '<:raw:encoding(utf16)', $csv_file )
      or die "Cannot read $csv_file: $!\n";

[download]

Doing the same, but using utf8 instead of uft16 didn't brought the expected result (no error).

Do you have any hint about that? Should I write specific code to remove the BOM from the beginning of the file? I don't need it anyway.

Thanks,

Alceu Rodrigues de Freitas Junior
---------------------------------
"You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill

In reply to Problems reading UTf-8 file with BOM by glasswalk3r

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.