lbova has asked for the wisdom of the Perl Monks concerning the following question:

Please forgive the cross-post. I asked this question originally on perl.beginners, because that's what I am.
But since I haven't had any feedback, I thought I'd ask here ...

I work with utf8 non-English text frequently, using perl 5.8.8 on Solaris-10, using the enUS-utf8 locale
and the dtterm in the CDE operating environment. In this terminal, I can cat, grep, sort, vim, etc. with utf8 text
perfectly well. I can include the following lines in my code and work successfully
( I'm writing these lines from memory, so please forgive my syntax.): 

binmode STDIN, ':utf8';
binmode STDOUT, ':utf8';
binmode STDERR, ':utf8';
use utf8;

[download]
This allows my apps to work gracefully with utf8 data. However, when I use the debugger (perl -d),
I have always had some problems. Basically, I could see the utf8 literals in my code, but anytime
I "p" or "x", the characters above the low ASCII range simply disappeared.
They were still in the variables, which I could tell by counting the length of strings, but they
vanished from the visible output.

Recently, I found a clue to this problem, (in the book "Pro Perl Debugging")
and now I include the following in my app: 

binmode $DB::OUT, ':utf8';

[download]
Eureka! Now the utf8 data is visible when I "p" or "x". This is a great improvement.
However, now the utf8 literals in my code are mangled. They display with some form of "^_"
instead of displaying as "themselves".

I am a lot better off this way than I used to be, but I'm sure there's
some more magic to be applied to this problem. As a beginner, I spend
a lot of time in the debugger, so a solution would be most helpful.

Thanks,
Lou

Replies are listed 'Best First'.
Re: Displaying utf8 text in perl -d
by Joost (Canon) on Sep 16, 2007 at 03:08 UTC
    I can't reproduce your problem.

    this works for me: 

    $ perl -d -e'binmode DB::OUT,":utf8"'

Loading DB routines from perl5db.pl version 1.28
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(-e:1):   binmode DB::OUT,":utf8"
  DB<1> 

  DB<1> s
Debugged program terminated.  Use q to quit or R to restart,
  use o inhibit_exit to avoid stopping after program termination,
  h q, h R or h o to get additional info.  
  DB<1>  p "\x{224}"

    [download]
    Ȥ

    (I had to put the printed char outside of the code tags because perlmonk's code mangles it otherwise)

    update: note that if the binmode() part of the code hasn't (yet) been executed, I get the usual error messages: 

      DB<1> p "\x{224}"
Wide character in print at (eval 5)[/usr/local/lib/perl5/5.8.8/perl5db
+.pl:628] line 2.
 at (eval 5)[/usr/local/lib/perl5/5.8.8/perl5db.pl:628] line 2
        eval '($@, $!, $^E, $,, $/, $\\, $^W) = @saved;package main; $
+^D = $^D | $DB::db_stop;
print {$DB::OUT}  "\\x{224}";

;' called at /usr/local/lib/perl5/5.8.8/perl5db.pl line 628
        DB::eval called at /usr/local/lib/perl5/5.8.8/perl5db.pl line 
+3410
        DB::DB called at -e line 1

    [download]
    Update 2:

    You should be using binmode DB::OUT,":utf8" instead of binmode $DB::OUT,":utf8"

    "What should it profit a man, if he should win a flame war, yet lose his cool?"
[reply]
[d/l]
[select]
Re: Displaying utf8 text in perl -d
by graff (Chancellor) on Sep 16, 2007 at 03:55 UTC
    I've had a lot of nasty trouble using "perl -d" with scripts that need to handle utf8 text. I'll need to try the binmode $DB::OUT, ':utf8' trick to see if that helps.

    What I have tried, which has usually had some positive effect on interactive debugging sessions, is to manually repeat the binmode ..., ":utf8" for both STDOUT and STDERR as commands to the debugger after it starts, and then always use "print $var" or ("print Dumper $ref" when using Data::Dumper) rather than just the "p" debugger command; that way, I don't get "Wide character in print" warnings.

    Other problems I've had involve trying to step past regex operations on wide-character strings, and suddenly getting an "out-of-memory" crash (sometimes phrased as "ridiculous memory request" or words to that effect).

    There seems to be some sort of bad interaction between the debugger and the wide-character handling in regexes; I've stumbled over it on macosx/perl 5.8.6, and on freebsd/perl 5.8.7/8 (there seems to be somewhat less trouble with 5.8.8, but I've still seen some problems there).

    I really should try to put together a simple case that demonstrates the trouble, and report a bug...

[reply]
[d/l]
[select]
      I really should try to put together a simple case that demonstrates the trouble, and report a bug...

      For any monk who regularly, or even intermittently, uses a Win32 (native) build of Perl, your posts (at least 50% of them) should be required reading.

      I've never used anything but a win32 version, AS or home-built, version of Perl, and I have learnt more from your posts that any single other source.,

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

Back to Seekers of Perl Wisdom