comment on

The output of the script is in the original post:

Ein <#c3><#96>konomisches Modell
Ein <#d6>konomisches Modell 1
Ein <#d6>konomisches Modell 1
[download]

I expected to see <#c3><#96> in all three cases. The actual confusion on the test script turns out to be a bug in the "print_chrcodes" function. I either should have used Devel::Peek, or ensured the function used "use bytes":

sub print_chrcodes
{
    my $str = shift;
    my $ret;
    use bytes;  #<<<<<<<<<here
    foreach my $ascval (unpack("C*", $str)) {
        if($ascval == 13) {
            $ret .= '<cr>';
            next;
        }
        if($ascval == 10) {
            $ret .= '<nl>';
            next;
        }
        if($ascval <  128) {
            $ret .= chr($ascval);
            next;
        }
        $ret .= sprintf ("<#%x>",$ascval) if($ascval >= 128);
    }
    return $ret;
}
[download]

What I was diagnosing was a situation where Unicode did not cleanly go through Storable::nfreeze, a database, then Storable::thaw :

$hash->{TITLE}="f\xc3\xa4ce=\xe2\x98\xbb";
print "TITLE=$hash->{TITLE}\n";
my $nfreeze = Storable::nfreeze($hash);
my $obj     = Storable::thaw($nfreeze);
print '${^UNICODE}='.${^UNICODE}."\n";
print "TITLE=$obj->{TITLE}\n";
[download]

That eventually resolved itself. The setting of PERL_UNICODE=SAD vs. PERL_UNICODE=0 was masking the true problem, as it allowed certain sequences perl thought of as latin1 to be seen on the terminal as their equivalent utf-8 character (in my test case, black smiling face ☻).

So in the end with: use encoding "utf-8-strict"; added to the script, and PERL_UNICODE=SAD, and proper terminal settings, and Apache Accept-Charset, and SET CLIENT-ENCODING UTF8 I'm round tripping Uncode through the entire Browser->Apache->Perl->Database->Perl->Apache->Browser System.

Thanks for the help.

In reply to Re^2: Why does perl's internal utf8 seem to allow single-byte latin1? by brycen
in thread Why does perl's internal utf8 seem to allow single-byte latin1? by brycen

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


go ahead... be a heretic
	PerlMonks