I have a big application, all written in Perl. It uses a MySQL database and saves in some cases Perl structures to the database. This Perl structures are dumped with Data::Dumper and restored with eval.

Everyting is in Unicode: program sources, data files, database. Each program and Module begins with use utf8. Non-ASCII Unicode data is handled without problems, except in one case: data dumped with Data::Dumper, then restored and then written to the database (now without Dumper). Such data is corrupted.

After some research i finally found out that the data restored with eval looses its utf-8 bit. All bytes are correct, but Perl does not know that it is utf8-encoded. DBI then fails when it tries to include such data in SQL statements.

Unluckilly the application is quite big, What is the simplest way to fix this problem? All I need is a way to tell Perl in evals, that string which are generated by the eval are in Unicode (Useqq does not solve it)

This program demonstrates the problem:

#!/usr/bin/perl -w
use strict;
use utf8;

use Data::Dumper;
# $Data::Dumper::Useqq = 1;

binmode STDOUT, 'utf8';

our $VAR1;
my $data = 'ä'; # this is a non-ACII a Umlaut

my $dump = Dumper( $data );

eval $dump;

if ( $data eq $VAR1 ) {
    print " == equal\n";
}
else {
    print " != not equal\n";
}
print $dump, "\n";
print Dumper( $VAR1 ), "\n";

print "original is utf8 = '" . utf8::is_utf8( $data ) . "'\n";
print "restored is utf8 = '" . utf8::is_utf8( $VAR1 ) . "'\n";

[download]

Output is

 == equal
$VAR1 = "\x{e4}";

$VAR1 = 'ä';

original is utf8 = '1'
restored is utf8 = ''

[download]
PS: it is on a Mac OSX 10.5.8 with Perl v5.8.8 built for darwin-thread-multi-2level

In reply to Data::Dumper and utf8 by thedi

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.