So, I know it's all confusing. Took me forever. But it's actually really simple. A string of bytes is nothing. It's just binary data. You have to know what it's supposed to be and tell your code when coming from binary and going back to it. The raw stuff doesn't know (well, some charsets do have BOM flags but it's not something on which you can rely here). Your DBI/DBD driver can do the encode/decode two-step for you automatically as I suggested (might work even if table definition is wrong but it's best to ensure it's in agreement). :P Examples of the setting to check include–

• DBD::mysql -> mysql_enable_utf8
  • This attribute determines whether DBD::mysql should assume strings stored in the database are utf8. This feature defaults to off.
• DBD::SQLite -> sqlite_unicode
  • If the attribute $dbh->{sqlite_unicode} is set, strings coming from the database and passed to the collation function will be properly tagged with the utf8 flag; but this only works if the attribute is set before the first call to a perl collation sequence . The recommended way to activate unicode is to set the sqlite_unicode parameter at connection time
• DBD::Pg -> pg_enable_utf8 (integer)
  • DBD::Pg specific attribute. The behavior of DBD::Pg with regards to this flag has changed as of version 3.0.0. The default value for this attribute, -1, indicates that the internal Perl utf8 flag will be turned on for all strings coming back from the database if the client_encoding is set to 'UTF8'. Use of this default is highly encouraged. If your code was previously using pg_enable_utf8, you can probably remove mention of it entirely. :\

Update: s/simply/simple/;

See Re: A puzzled garbage character issues by use Perl-DBI for DBD::ODBC, DBD::Oracle, and unicode round-trip tests.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Thanks for the info. Man, this is a PITA :S Think I may have to take a break, and come back to it tomorrow.

There is definitely something up - because even using basic DBI connection, it still messes it up:



    my $dsn = "DBI:mysql:database=$db_cfg->{database};host=$db_cfg->{h
+ost};port=3307";
    my $dbh = DBI->connect($dsn, $db_cfg->{login}, $db_cfg->{password}
+);

    $dbh->{mysql_enable_utf8} = 1;

    my $sth = $dbh->prepare( "INSERT INTO ReadingGrabCache SET title =
+ ?" );
       $sth->execute( $title ) or die $DBI::errstr;
[download]

Eugh :/

Did you know you're not checking the status of the connect or the prepare? And if you turn on RaiseError in the connect, you'll automatically check all DBI methods, and you won't even need the 'or die' on the execute.

$title = encode("utf-8",$title);
[download]

Why do you do that?

Sorry, that was a mistake :) This is what I have:

        print "GOT TITLE BEFORE: $title \n<Br><Br>";

        $DB->table("ReadingGrabCache")->add( { title => $title, url =>
+ "Foo" });

        my $test = $DB->table("ReadingGrabCache")->select ( { url => "
+Foo" })->fetchrow_hashref;

        print "BEFORE RE-ENCODING: $test->{title} \n<br>";

        $test->{title} = decode("utf-8",$test->{title});

        print "AFTER RE-ENCODING: $test->{title} \n<br>";
[download]

GOT TITLE BEFORE: Румыния не будет участвовать в �Евровидении-2016� из-за денег - Газета.Ru

BEFORE RE-ENCODING: (ie right out of the DB grab) � �ƒм�‹ния не б�ƒде�‚ �ƒ�‡ас�‚вова�‚�Œ в «�•в�€овидении-2016» из-за денег - �“азе�‚а.Ru
AFTER RE-ENCODING: Румыния не будет участвовать в �Евровидении-2016� из-за денег - Газета.Ru

that's because you binmoded STDOUT... here's a simpler way to show the problem:

perl -E'binmode STDOUT, ":utf8"; say "Румыния";'