Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

A Win32 new Perl build dilemma

by mikegold10 (Novice)
on Nov 29, 2021 at 01:46 UTC ( #11139195=perlquestion: print w/replies, xml ) Need Help??

mikegold10 has asked for the wisdom of the Perl Monks concerning the following question:

I just built a version of the latest version of Perl, v5.35.6, from source and am having an odd issue with the output of the ASCII Escape character when it is printed. When I run a script at the Windows 10 command prompt, Perl, instead of dumping the Escape character as a literal character into the console when instructed to do so, prints a left pointing arrow (probably the printable glyph representation of the Escape character). Before I present a code example, I should add for completeness sake that I am in the default Windows code page of 437. (Verified via chcp, which reports: Active code page: 437 . Playing with the code page (e.g., changing it to 65001 or 1252) doesn't help, other than to make the left-pointing-arrow as shown below into a non-printable "box" character representation, probably because the glyph representation of Escape being printed by Perl as a left pointing arrow is invalid for those two code pages or whatever is coming out is missing from the characters in the Consolas font I am using for my Command Prompt). Here is an example: (Clarification: a pair of double-quotes is used to cause double-quotes to get escaped for proper cmd.exe interpretation):

perl.exe -e "print ""this \x1B[1;31mred\x1B[0m text\n"" "
Output (Incorrect - Perl is changing the chr(27) that precedes each left bracket i.e., the Escape ASCII character, into a left arrow):
this ←[1;31mred←[0m text
What I am trying to do is to use an ANSI Escape Code above to change the color of the text to red for the word "red" (i.e., <Esc>[1;31m ==> Set foreground color to 31 Red and <Esc>[0m ==> Reset all modes/styles). For some bizarre reason, Perl is causing a character U+2190, best I can tell, which is the Unicode representation of a left-arrow (a glyph that is sometimes used to represent the <Esc> character as a printable character) to be output instead of the proper Escape ASCII character I am expecting. There is nothing wrong with the code, mind you, because if I do the following:
perl.exe -e "print ""this \x1B[1;31mred\x1B[0m text\n"" " | cat
Output (correct):
this red text
So, it seems like Perl is treating a redirection of STDOUT differently and actually sending a proper escape character. I also tried the following (redirecting output to a file and outputting said file, instead of piping to cat) to see what happens:
perl.exe -e "print ""this \x1B[1;31mred\x1B[0m text\n"" " >tmp.txt & t +ype tmp.txt
Output (correct):
this red text
Just to make absolutely certain that the issue is not with the code or some misbehavior of the windows command prompt (i.e., c:\windows\system32\cmd.exe), I ran the code through a different Perl interpreter, the one that comes with Cygwin, to see how it would behave in this case:
c:\cygwin64\bin\perl5.32.1.exe -e "print ""this \x1B[1;31mred\x1B[0m t +ext\n"" "
Output (correct):
this red text
Perhaps to no surprise, the Cywin Perl interpreter did the right thing, even though its output was not in any way redirected. Here is what we can rule out, right off the bat:
  • It is not a Perl 5.35.x issue, because I have an old 5.34 build that behaves in the same way (built with the same build settings/defines).
  • As demonstrated via type/cat/Cygwin Perl, it is not a Windows Command Prompt (i.e., cmd.exe) or active code page issue, either.
I am not sure what the true cause is, but I believe that the problem may somehow stem from my build config. So let me present it next by using perl -V. I used Visual C++ 2022 as my compiler, for this build and used Visual C++ 2019 for the Perl 5.34 build that behaves in the same way with regard to the Escape character handling:
C:\>perl.exe -V Summary of my perl5 (revision 5 version 35 subversion 6) configuration +: Platform: osname=MSWin32 osvers=10.0.19043.1348 archname=MSWin32-x64-multi-thread-64int uname='' config_args='undef' hint=recommended useposix=true d_sigaction=undef useithreads=define usemultiplicity=define use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define Compiler: cc='cl' ccflags ='-nologo -GF -W3 -MD -DWIN32 -D_CONSOLE -DNO_STRICT -DNDE +BUG -DWIN64 -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D +_WINSOCK_DEPRECATED_NO_WARNINGS -DPERL_TEXTMODE_SCRIPTS -DMULTIPLICI +TY -DPERL_IMPLICIT_SYS -DWIN32_NO_REGISTRY' optimize='-O1 -Ob2 -Oi -Oy -GL -Gm- -GS- -Gy -sdl- -Zc:inline -Zi +-fp:precise -arch:AVX512 -fp:precise' cppflags='-DWIN32' ccversion='19.31.30818' gccversion='' gccosandvers='' intsize=4 longsize=4 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=undef longlongsize=8 d_longdbl=define longdblsize=8 longdblkind=0 ivtype='__int64' ivsize=8 nvtype='double' nvsize=8 Off_t='__int64' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='link' ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpat +h:"c:\perl\5.35.6\lib\CORE" -machine:AMD64 -subsystem:console,"6.00" +' libpth="c:\Program Files\Microsoft Visual Studio\2022\Preview\VC\T +ools\MSVC\14.31.30818\\lib\x64" libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib +comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi3 +2.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib o +dbccp32.lib comctl32.lib msvcrt.lib vcruntime.lib ucrt.lib perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.l +ib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib net +api32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.l +ib odbccp32.lib comctl32.lib msvcrt.lib vcruntime.lib ucrt.lib libc=ucrt.lib so=dll useshrplib=true libperl=perl535.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs dlext=dll d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg - +libpath:"c:\perl\5.35.6\lib\CORE" -machine:AMD64 -subsystem:console, +"6.00"' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_NO_REGISTRY USE_PERLIO USE_PERL_ATOF USE_THREAD_SAFE_LOCALE Built under MSWin32 Compiled at Nov 27 2021 18:54:37 @INC: c:/perl/5.35.6/lib
Just to compare and contrast, here is what the Perl from Cygwin has as its configuration:
c:\>perl5.32.1.exe -V Summary of my perl5 (revision 5 version 32 subversion 1) configuration +: Platform: osname=cygwin osvers=3.2.0(0.34053) archname=x86_64-cygwin-threads-multi uname='cygwin_nt-10.0 cygwinpro 3.2.0(0.34053) 2021-03-29 08:42 x8 +6_64 cygwin ' config_args='-des -Dprefix=/usr -Dmksymlinks -Darchname=x86_64-cyg +win-threads -Dlibperl=cygperl5_32.dll -Dcc=gcc -Dld=g++ -Accflags=-gg +db -O2 -pipe -Wall -Werror=format-security -D_FORTIFY_SOURCE=2 -fstac +k-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/mnt/ +share/cygpkgs/perl/perl.x86_64/build=/usr/src/debug/perl-5.32.1-2 -fd +ebug-prefix-map=/mnt/share/cygpkgs/perl/perl.x86_64/src/perl-5.32.1=/ +usr/src/debug/perl-5.32.1-2 -fwrapv' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE - +ggdb -O2 -pipe -Wall -Werror=format-security -D_FORTIFY_SOURCE=2 -fst +ack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/mn +t/share/cygpkgs/perl/perl.x86_64/build=/usr/src/debug/perl-5.32.1-2 - +fdebug-prefix-map=/mnt/share/cygpkgs/perl/perl.x86_64/src/perl-5.32.1 +=/usr/src/debug/perl-5.32.1-2 -fwrapv -fno-strict-aliasing' optimize='-O3' cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE - +ggdb -O2 -pipe -Wall -Werror=format-security -D_FORTIFY_SOURCE=2 -fst +ack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/mn +t/share/cygpkgs/perl/perl.x86_64/build=/usr/src/debug/perl-5.32.1-2 - +fdebug-prefix-map=/mnt/share/cygpkgs/perl/perl.x86_64/src/perl-5.32.1 +=/usr/src/debug/perl-5.32.1-2 -fwrapv -fno-strict-aliasing' ccversion='' gccversion='10.2.0' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='g++' ldflags =' -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,- +-enable-auto-image-base -fstack-protector-strong' libpth=/usr/lib libs=-lpthread -lnsl -lgdbm -ldb -ldl -lcrypt -lgdbm_compat perllibs=-lpthread -lnsl -ldl -lcrypt libc=/usr/lib/libcygwin.a so=dll useshrplib=true libperl=cygperl5_32.dll gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=dll d_dlsymun=undef ccdlflags=' ' cccdlflags=' ' lddlflags=' --shared -Wl,--enable-auto-import -Wl,--export-all-sy +mbols -Wl,--enable-auto-image-base -fstack-protector-strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API USE_THREAD_SAFE_LOCALE Locally applied patches: Cygwin: README Cygwin: use auto-image-base instead of fixed DLL base address Cygwin: modify hints Cygwin: Configure correct libsearch Cygwin: Configure correct libpth Cygwin: Encode fix for CVE-2021-36770 Cygwin: Win32 correct UTF8 handling Built under cygwin Compiled at Aug 14 2021 09:07:12 %ENV: CYGWIN="glob:ignorecase winsymlinks:native pipe_byte" @INC: /usr/local/lib/perl5/site_perl/5.32/x86_64-cygwin-threads /usr/local/share/perl5/site_perl/5.32 /usr/lib/perl5/vendor_perl/5.32/x86_64-cygwin-threads /usr/share/perl5/vendor_perl/5.32 /usr/lib/perl5/5.32/x86_64-cygwin-threads /usr/share/perl5/5.32

Replies are listed 'Best First'.
Re: A Win32 new Perl build dilemma
by afoken (Canon) on Nov 29, 2021 at 07:39 UTC
    Output (Incorrect - Perl is changing the chr(27) that precedes each left bracket i.e., the Escape ASCII character, into a left arrow):

    Don't blame perl. Character 27 in code page 437 is the left arrow, nothing is changed. The fact that you see the arrow indicates that no ANSI interpretion happens for STDOUT. That's not uncommon, DOS did not interpret ANSI sequences by default, and neither did Windows. Both never needed to do so, because both DOS and Windows never ran natively on terminals. In DOS, you needed to load ANSI.SYS or similar, Windows needs some other changes, and things have changed again in some Windows 10 update. See https://stackoverflow.com/questions/16755142/ and pages linked from there, and https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences for Windows 10 changes.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      I followed the links in your post reading everything carefully. A comment to one of the StackOverflow questions had an easy solution to the problem, as follows (in case anyone else runs into this):
      Add the following value of type DWORD to the registry at the indicated key:

      HKEY_CURRENT_USER\Console VirtualTerminal = 1
      Problem solved and thank you! I spent several hours trying to figure this one out...
Re: A Win32 new Perl build dilemma
by Corion (Patriarch) on Nov 29, 2021 at 07:25 UTC

    I don't use Cygwin, but I guess that all Cygwin programs run their output through an ANSI escape interpreter, while the Win32 Perl does not.

    This would explain why you see the ANSI escapes interpreted with Cygwin Perl but raw with the Win32 Perl.

    You can "fix" this by loading the module Win32::Console::ANSI in your program, which interprets the output as ANSI and instructs the Windows console to render it properly. Of course that means you should only load that module on "real" Win32 Perl, not Cygwin Perl.

    use if $^O eq 'MSWin32', 'Win32::Console::ANSI';
      Win32::Console::ANSI did not resolve the issue:
      C:\>perl -e "use warnings; use strict; use Win32::Console::ANSI; print ""\e[1;34mThis text is bold blue.\e[0m\n""; "
      ←[1;34mThis text is bold blue.←[0m
      
Re: A Win32 new Perl build dilemma
by choroba (Archbishop) on Nov 29, 2021 at 09:33 UTC
    > v5.35.6

    Are you aware that odd minor versions are development versions? The latest stable Perl version is 5.34.0.

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
      I am aware and the issue is present in version 5.34.0, as I pointed out in my original question.
Re: A Win32 new Perl build dilemma
by aitap (Curate) on Nov 29, 2021 at 07:20 UTC
    Try using WinAPIOverride to see if there's any difference in WinAPI calls between the old and the new Perl? If there isn't any, it must be somewhere else (flags in the EXE itself set by the linker? something fishy in the manifest?). If you do find something, it could be traced to the CRT and Perl's use of it.
      Turned out to be a need for a registry change, as mentioned in my reply to the first post...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11139195]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (3)
As of 2022-08-09 05:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?