Re: DBD::Pg encodes Perlstring to UTF-8 bytes instead of WIN1252 regardless client encoding

I'm going to describe how DBD::mysql works. I suspect DBD::Pg works the same way.

Perl has two ways of storing strings. DBI or DBD::mysql looks at the internal buffer of scalars without checking which storage format was used, so every time you pass a string, it's as if you actually passed

use Encode qw( is_utf8 encode_utf8 );

is_utf8($string) ? encode_utf8($string) : $string
[download]

This is a bug, but it almost always does the right thing.

Workaround:

If you have a decoded string (a string of Unicode code points), you can use the following:

use Encode qw( encode_utf8 );

$dbh->do("SET NAMES utf8");

my $sth = ...;
$sth->execute(encode_utf8($decoded));
[download]

If you have a string encoded using cp1252, you can use the following:

use Encode qw( decode );

$dbh->do("SET NAMES utf8");

my $sth = ...;
$sth->execute(decode('cp1252', $encoded));
[download]

If you have a string encoded using cp1252 you want to avoid any encoding and decoding on the Perl side, you can use the following:

sub _d { my ($s) = @_; utf8::downgrade($_); $s }

$dbh->do("SET NAMES cp1252");

my $sth = ...;
$sth->execute(_d($encoded));
[download]

Notes:

Passing mysql_enable_utf8=>1 to DBI->connect does $dbh->do("SET NAMES utf8"); for you. Later changes to mysql_enable_utf8 does not.
is_utf8 always returns true for strings returned by Encode::decode and Encode::decode_utf8.
is_utf8 always returns false for strings returned by Encode::encode, Encode::encode_utf8 and Encode::from_to.

Comment on Re: DBD::Pg encodes Perlstring to UTF-8 bytes instead of WIN1252 regardless client encoding Select or Download Code

Replies are listed 'Best First'.
Re^2: DBD::Pg encodes Perlstring to UTF-8 bytes instead of WIN1252 regardless client encoding by mje (Curate) on Feb 04, 2014 at 11:06 UTC
I don't use DBD::mysql these days but I do maintain DBD::ODBC and help maintain DBD::Oracle. I'm not aware of anything in DBI which "tampers" in any way with strings passed to the prepare or bind_param methods. I've looked at DBD::mysql code and I cannot see anything that explains your "it's as if you actually passed". The original poster said his inserts were with the do method and no bound params and as far as I can see if he'd done that with DBD::mysql (instead of postgres) the string would have ended up in the mysql client API mysql_stmt_prepare untouched. Are you saying bound parameters work differently in DBD::mysql? Other than some rather strange enbabling/disabling of SET NAMES in the code I can only find a few calls to sv_utf8_decode to decode UTF8 data received from mysql server and a test of: `if (SvUTF8(str)) SvUTF8_on(result);` [download] in the quote method. I'm really interested in this from the point of another DBD maintainer.	[reply] [d/l]
Re^3: DBD::Pg encodes Perlstring to UTF-8 bytes instead of WIN1252 regardless client encoding by ikegami (Patriarch) on Mar 27, 2014 at 19:10 UTC
I've looked at DBD::mysql code and I cannot see anything that explains your "it's as if you actually passed". Does it use `SvPV(sv)` without checking separate handling based on `SvUTF8(sv)`? It might do by using the default `char` typemap (e.g. `void xsfunc(char s) { ... }`). As you can see, the following C function is equivalent to that code: use strict; use warnings; use Test::More; use Encode qw( is_utf8 encode_utf8 ); use Inline C => <<'__EOC__'; void candidate(SV* sv) { dXSARGS; STRLEN len; char* buf = SvPV(sv, len); SP[0] = sv_2mortal(newSVpvn(buf, len)); XSRETURN(1); } __EOC__ sub baseline { is_utf8($_[0]) ? encode_utf8($_[0]) : $_[0] } sub _u { my ($s) = @_; utf8::upgrade($s); $s } sub _d { my ($s) = @_; utf8::downgrade($s); $s } sub printable { sprintf("%v04X", $_[0]) } my @tests = ( [ '00-7F', "a" ], [ '80-FF,UTF8=0', _d(chr(0xE9)) ], [ '80-FF,UTF8=1', _u(chr(0xE9)) ], [ '>FF', chr(0x2660) ], ); plan tests => 0+@tests; for (@tests) { my ($test_name, $input) = @$_; my $got = candidate($input); my $expected = baseline($input); #is($got, $expected, $test_name); is(printable($got), printable($expected), $test_name); } [download] `1..4 ok 1 - 00-7F ok 2 - 80-FF,UTF8=0 ok 3 - 80-FF,UTF8=1 ok 4 - >FF` [download] I'm really interested in this from the point of another DBD maintainer. Using the same configuration for all tests, can you roundtrip all of the strings mentioned earlier (as verified by `is` or `eq`)? `my @tests = ( [ '00-7F', "a" ], [ '80-FF,UTF8=0', _d(chr(0xE9)) ], [ '80-FF,UTF8=1', _u(chr(0xE9)) ], [ '>FF', chr(0x2660) ], ); plan tests => 0+@tests; my $dbh = ... connect and setup as you wish ... my $sth = $dbh->prepare('SELECT ?'); for (@tests) { my ($test_name, $input) = @$_; $sth->execute($input) or die; my $row = $sth->fetch() or die; $sth->finish() or die if $row; my $got = $row->[0]; is(printable($got), printable($input), $test_name); }` [download]	[reply] [d/l] [select]