Problem with special characters in a csv file

shilpam has asked for the wisdom of the Perl Monks concerning the following question:

I am having trouble while printing some special characters in a csv file. I have written a script which runs a sql query, gets the result from database and prints the value in a csv file. I am using Test::CSV_XS. The script is working fine except when it encounters data like:-
The value stored in the database is First Encounter®. I am getting the values from the database and then printing it into a csv file. But, whenever it encounters the above mentioned value, the Perl script gives following error:
combine() failed on argument: First Encounter®
Can there be a solution wherein the data which is stored in the database is "First Encounter&# 174". But, when it has to print in the csv file, it converts this data into First Encounter®?
I used HTML::Entities to convert "First Encounter&# 174" to First Encounter®. But, when I try to print the value in a csv file, it gives me the same (above mentioned) error. Any idea how to get these characters displayed correctly in a csv file?

Comment on Problem with special characters in a csv file

Replies are listed 'Best First'.
Re: Problem with special characters in a csv file by jZed (Prior) on Jul 22, 2004 at 05:11 UTC
Set binary=1 in Text::CSV_XS which will allow it to accept "special" characters. Also, since you are going from a database to CSV, you might want to use DBD::CSV which provides a DBI interface to Text:::CSV_XS.	[reply]
Re: Problem with special characters in a csv file by matija (Priest) on Jul 22, 2004 at 05:17 UTC
One possible solution is to open CSV something like this: `my $csv = Text::CSV_XS->new ({'always_quote' => 1, 'eol' => "\n", 'binary' => 1, });` [download] That will allow "funny characters" directly in the CSV file. If you are determined to change the characters to entities, you could do it something like this: `$string=~s/[\x7f-\xff\&]/"&#".ord($1).";"/ge;` [download]	[reply] [d/l] [select]
Re^2: Problem with special characters in a csv file by shilpam (Sexton) on Jul 22, 2004 at 08:54 UTC
Thanks a lot! The solution worked!!!	[reply]
Re^2: Problem with special characters in a csv file by Anonymous Monk on Apr 16, 2011 at 08:10 UTC
I had the same problem with csv files, but i solved it with saving my Excel file to txt and then just replacing (with replace all)tabs with comas, and I used the txt file for the print merge.	[reply]