in reply to Re^2: My PerlTidy utility is choking on non-ASCII characters, help me figure out why?
in thread My PerlTidy utility is choking on non-ASCII characters, help me figure out why?

oh, right, forgot about that

What about the Dumper?

If i try 

perl -CSD -le " print qq{use utf8;\nprint qq{I \x{2665} Perl\n};\n}; "
+ >foo.pl

[download]

I get 

use utf8;
print qq{I ♥ Perl
};

If I run that through perltidy or your program its unchanged

cmd.exe doesn't know how to display the unicode without a chcp, but the bytes are the same

If I add to your program 

    use Data::Dump qw/pp /;
    print STDERR pp("$output_string");

[download]

I get unchanged correct result as expected 

"use utf8;\nprint qq{I \xE2\x99\xA5 Perl\n};\n\n"

[download]

by default cmd.exe does not unicode 

$ chcp
Active code page: 437

$ type foo.tdy
use utf8;
print qq{I ΓΦΡ Perl
};

If I change it I get a heart 

$ chcp 65001
Active code page: 65001

$ type foo.tdy
use utf8;
print qq{I ♥ Perl
};

Replies are listed 'Best First'.
Re^4: My PerlTidy utility is choking on non-ASCII characters, help me figure out why?
by Cody Fendant (Hermit) on Aug 29, 2016 at 05:38 UTC

    I must be misunderstanding something, but I don't know what you're suggesting I do to actually make my code work.

    How would I use that to make a replacement for my Perl Tidy utility?

[reply]

      I must be misunderstanding something, but I don't know what you're suggesting I do to actually make my code work. How would I use that to make a replacement for my Perl Tidy utility?

      I've been suggesting number 4 from Basic debugging checklist, examine the data with Data::Dumper/Data::Dump/JSON/YAML

      I tried to reproduce your problem,

      First I created a perl file written in unicode called foo.pl

      Then I gave this file to perltidy, your perltidy, your perltidy with Dump ... in all versions the non-ascii characters survived unharmed :)

      I also showed how cmd.exe can appear to show "corruption" of the file, but the actual bytes are as they should be

      Text editors can show the same corruption by reading a utf-8 file as some other encoding

      Either BBEdit is treating these bytes as something other than utf-8

      Or Perl::Tidy is producing the wrong bytes

      I think I've shown Perl::Tidy is doing its job without problems, what do you think?

[reply]
[d/l]

In Section Seekers of Perl Wisdom