Re: Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV

To get things started, I was having major issues until I changed your two instances of encoding(UTF-8) to utf8.

Next, are you sure you want \x{feff}? When I get rid of it, I start getting things quoted:

Row 1, Col 1 Is Quoted:  Row1Col1
Row 1, Col 2 Is Quoted:  Row1Col2
Row 1, Col 3 Is Quoted:  Row1Col3
Row 2, Col 1 Is Not Quoted:  Row2Col1
Row 2, Col 2 Is Not Quoted:  Row2Col2
Row 2, Col 3 Is Not Quoted:  Row2Col3
Row 3, Col 1 Is Not Quoted:  Row3Col1
Row 3, Col 2 Is Quoted:  Row3Col2
Row 3, Col 3 Is Quoted:  Row3Col3
Row 4, Col 1 Is Quoted:  Row4Col1
Row 4, Col 2 Is Not Quoted:  Row4Col2
Row 4, Col 3 Is Quoted:  Row4Cşol3
Row 6, Col 1 Is Not Quoted:  Row5Col1
Row 6, Col 2 Is Not Quoted:  Row5Col2
Row 6, Col 3 Is Quoted:  Row5
Col3
[download]

also, it gets rid of the weird ş characters. One more thing, your input file isn't formatted consistently... There are some /n and double \x{fe}\x{fe} characters where I don't think you intended them to be.

Comment on Re: Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS Select or Download Code

Replies are listed 'Best First'.
Re^2: Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS by Jim (Curate) on Jul 10, 2012 at 17:43 UTC
Thank you, frozenwithjoy, for your reply. I appreciate it. To get things started, I was having major issues until I changed your two instances of encoding(UTF-8) to utf8. My use of `:encoding(UTF-8)` instead of `:encoding(utf8)` is intentional and purposeful. Next, are you sure you want \x{feff}? I'm absolutely sure. It's the Unicode byte order mark, and it's vital to my test because the real CSV files I must process invariably have the byte order mark in them, encoded in UTF-8 (`EF BB BF`). When I get rid of it, I start getting things quoted: … also, it gets rid of the weird ş characters. This doesn't happen for me. I get the same unhelpful results regardless of the presence or absence of the Unicode byte order mark. What versions of Perl and Text::CSV_XS are you using? One more thing, your input file isn't formatted consistently... There are some /n and double \x{fe}\x{fe} characters where I don't think you intended them to be. Those literal newlines and Latin small thorns (þ) are intentional and purposeful. They're vital to my test. Jim	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: Non-ASCII Characters As Separators/Quotes/Escapes When Using Text::CSV_XS
by Jim (Curate) on Jul 10, 2012 at 17:43 UTC

Thank you, frozenwithjoy, for your reply. I appreciate it.

To get things started, I was having major issues until I changed your two instances of encoding(UTF-8) to utf8.

My use of :encoding(UTF-8) instead of :encoding(utf8) is intentional and purposeful.

Next, are you sure you want \x{feff}?

I'm absolutely sure. It's the Unicode byte order mark, and it's vital to my test because the real CSV files I must process invariably have the byte order mark in them, encoded in UTF-8 (EF BB BF).

When I get rid of it, I start getting things quoted: … also, it gets rid of the weird ş characters.

This doesn't happen for me. I get the same unhelpful results regardless of the presence or absence of the Unicode byte order mark. What versions of Perl and Text::CSV_XS are you using?

One more thing, your input file isn't formatted consistently... There are some /n and double \x{fe}\x{fe} characters where I don't think you intended them to be.

Those literal newlines and Latin small thorns (þ) are intentional and purposeful. They're vital to my test.

Jim

[reply]
[d/l]
[select]