http://qs1969.pair.com?node_id=11139195

mikegold10 has asked for the wisdom of the Perl Monks concerning the following question:

I just built a version of the latest version of Perl, v5.35.6, from source and am having an odd issue with the output of the ASCII Escape character when it is printed. When I run a script at the Windows 10 command prompt, Perl, instead of dumping the Escape character as a literal character into the console when instructed to do so, prints a left pointing arrow (probably the printable glyph representation of the Escape character). Before I present a code example, I should add for completeness sake that I am in the default Windows code page of 437. (Verified via chcp, which reports: Active code page: 437 . Playing with the code page (e.g., changing it to 65001 or 1252) doesn't help, other than to make the left-pointing-arrow as shown below into a non-printable "box" character representation, probably because the glyph representation of Escape being printed by Perl as a left pointing arrow is invalid for those two code pages or whatever is coming out is missing from the characters in the Consolas font I am using for my Command Prompt). Here is an example: (Clarification: a pair of double-quotes is used to cause double-quotes to get escaped for proper cmd.exe interpretation):

perl.exe -e "print ""this \x1B[1;31mred\x1B[0m text\n"" "

[download]
Output (Incorrect - Perl is changing the chr(27) that precedes each left bracket i.e., the Escape ASCII character, into a left arrow):
this ←[1;31mred←[0m text
What I am trying to do is to use an ANSI Escape Code above to change the color of the text to red for the word "red" (i.e., <Esc>[1;31m ==> Set foreground color to 31 Red and <Esc>[0m ==> Reset all modes/styles). For some bizarre reason, Perl is causing a character U+2190, best I can tell, which is the Unicode representation of a left-arrow (a glyph that is sometimes used to represent the <Esc> character as a printable character) to be output instead of the proper Escape ASCII character I am expecting. There is nothing wrong with the code, mind you, because if I do the following:
perl.exe -e "print ""this \x1B[1;31mred\x1B[0m text\n"" " | cat

[download]
Output (correct):
this red text
So, it seems like Perl is treating a redirection of STDOUT differently and actually sending a proper escape character. I also tried the following (redirecting output to a file and outputting said file, instead of piping to cat) to see what happens:
perl.exe -e "print ""this \x1B[1;31mred\x1B[0m text\n"" " >tmp.txt & t
+ype tmp.txt

[download]
Output (correct):
this red text
Just to make absolutely certain that the issue is not with the code or some misbehavior of the windows command prompt (i.e., c:\windows\system32\cmd.exe), I ran the code through a different Perl interpreter, the one that comes with Cygwin, to see how it would behave in this case: 
c:\cygwin64\bin\perl5.32.1.exe -e "print ""this \x1B[1;31mred\x1B[0m t
+ext\n"" "

[download]
Output (correct):
this red text
Perhaps to no surprise, the Cywin Perl interpreter did the right thing, even though its output was not in any way redirected. Here is what we can rule out, right off the bat:
I am not sure what the true cause is, but I believe that the problem may somehow stem from my build config. So let me present it next by using perl -V. I used Visual C++ 2022 as my compiler, for this build and used Visual C++ 2019 for the Perl 5.34 build that behaves in the same way with regard to the Escape character handling: 
C:\>perl.exe -V
Summary of my perl5 (revision 5 version 35 subversion 6) configuration
+:

  Platform:
    osname=MSWin32
    osvers=10.0.19043.1348
    archname=MSWin32-x64-multi-thread-64int
    uname=''
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cl'
    ccflags ='-nologo -GF -W3 -MD -DWIN32 -D_CONSOLE -DNO_STRICT -DNDE
+BUG -DWIN64 -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -D
+_WINSOCK_DEPRECATED_NO_WARNINGS  -DPERL_TEXTMODE_SCRIPTS -DMULTIPLICI
+TY -DPERL_IMPLICIT_SYS -DWIN32_NO_REGISTRY'
    optimize='-O1 -Ob2 -Oi -Oy -GL -Gm- -GS- -Gy -sdl- -Zc:inline -Zi 
+-fp:precise -arch:AVX512 -fp:precise'
    cppflags='-DWIN32'
    ccversion='19.31.30818'
    gccversion=''
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=undef
    longlongsize=8
    d_longdbl=define
    longdblsize=8
    longdblkind=0
    ivtype='__int64'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='__int64'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='link'
    ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -libpat
+h:"c:\perl\5.35.6\lib\CORE"  -machine:AMD64 -subsystem:console,"6.00"
+'
    libpth="c:\Program Files\Microsoft Visual Studio\2022\Preview\VC\T
+ools\MSVC\14.31.30818\\lib\x64"
    libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  
+comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi3
+2.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib o
+dbccp32.lib comctl32.lib msvcrt.lib vcruntime.lib ucrt.lib
    perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.l
+ib  comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  net
+api32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.l
+ib odbccp32.lib comctl32.lib msvcrt.lib vcruntime.lib ucrt.lib
    libc=ucrt.lib
    so=dll
    useshrplib=true
    libperl=perl535.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg  -
+libpath:"c:\perl\5.35.6\lib\CORE"  -machine:AMD64 -subsystem:console,
+"6.00"'


Characteristics of this binary (from libperl):
  Compile-time options:
    HAS_TIMES
    HAVE_INTERP_INTERN
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_SYS
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_NO_REGISTRY
    USE_PERLIO
    USE_PERL_ATOF
    USE_THREAD_SAFE_LOCALE
  Built under MSWin32
  Compiled at Nov 27 2021 18:54:37
  @INC:
    c:/perl/5.35.6/lib

[download]
Just to compare and contrast, here is what the Perl from Cygwin has as its configuration:
c:\>perl5.32.1.exe -V
Summary of my perl5 (revision 5 version 32 subversion 1) configuration
+:

  Platform:
    osname=cygwin
    osvers=3.2.0(0.34053)
    archname=x86_64-cygwin-threads-multi
    uname='cygwin_nt-10.0 cygwinpro 3.2.0(0.34053) 2021-03-29 08:42 x8
+6_64 cygwin '
    config_args='-des -Dprefix=/usr -Dmksymlinks -Darchname=x86_64-cyg
+win-threads -Dlibperl=cygperl5_32.dll -Dcc=gcc -Dld=g++ -Accflags=-gg
+db -O2 -pipe -Wall -Werror=format-security -D_FORTIFY_SOURCE=2 -fstac
+k-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/mnt/
+share/cygpkgs/perl/perl.x86_64/build=/usr/src/debug/perl-5.32.1-2 -fd
+ebug-prefix-map=/mnt/share/cygpkgs/perl/perl.x86_64/src/perl-5.32.1=/
+usr/src/debug/perl-5.32.1-2 -fwrapv'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -
+ggdb -O2 -pipe -Wall -Werror=format-security -D_FORTIFY_SOURCE=2 -fst
+ack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/mn
+t/share/cygpkgs/perl/perl.x86_64/build=/usr/src/debug/perl-5.32.1-2 -
+fdebug-prefix-map=/mnt/share/cygpkgs/perl/perl.x86_64/src/perl-5.32.1
+=/usr/src/debug/perl-5.32.1-2 -fwrapv -fno-strict-aliasing'
    optimize='-O3'
    cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -D_GNU_SOURCE -
+ggdb -O2 -pipe -Wall -Werror=format-security -D_FORTIFY_SOURCE=2 -fst
+ack-protector-strong --param=ssp-buffer-size=4 -fdebug-prefix-map=/mn
+t/share/cygpkgs/perl/perl.x86_64/build=/usr/src/debug/perl-5.32.1-2 -
+fdebug-prefix-map=/mnt/share/cygpkgs/perl/perl.x86_64/src/perl-5.32.1
+=/usr/src/debug/perl-5.32.1-2 -fwrapv -fno-strict-aliasing'
    ccversion=''
    gccversion='10.2.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++'
    ldflags =' -Wl,--enable-auto-import -Wl,--export-all-symbols -Wl,-
+-enable-auto-image-base -fstack-protector-strong'
    libpth=/usr/lib
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lcrypt -lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lcrypt
    libc=/usr/lib/libcygwin.a
    so=dll
    useshrplib=true
    libperl=cygperl5_32.dll
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags=' --shared  -Wl,--enable-auto-import -Wl,--export-all-sy
+mbols -Wl,--enable-auto-image-base -fstack-protector-strong'


Characteristics of this binary (from libperl):
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_THREAD_SAFE_LOCALE
  Locally applied patches:
    Cygwin: README
    Cygwin: use auto-image-base instead of fixed DLL base address
    Cygwin: modify hints
    Cygwin: Configure correct libsearch
    Cygwin: Configure correct libpth
    Cygwin: Encode fix for CVE-2021-36770
    Cygwin: Win32 correct UTF8 handling
  Built under cygwin
  Compiled at Aug 14 2021 09:07:12
  %ENV:
    CYGWIN="glob:ignorecase winsymlinks:native pipe_byte"
  @INC:
    /usr/local/lib/perl5/site_perl/5.32/x86_64-cygwin-threads
    /usr/local/share/perl5/site_perl/5.32
    /usr/lib/perl5/vendor_perl/5.32/x86_64-cygwin-threads
    /usr/share/perl5/vendor_perl/5.32
    /usr/lib/perl5/5.32/x86_64-cygwin-threads
    /usr/share/perl5/5.32

[download]

Replies are listed 'Best First'.
Re: A Win32 new Perl build dilemma
by afoken (Chancellor) on Nov 29, 2021 at 07:39 UTC
    Output (Incorrect - Perl is changing the chr(27) that precedes each left bracket i.e., the Escape ASCII character, into a left arrow):

    Don't blame perl. Character 27 in code page 437 is the left arrow, nothing is changed. The fact that you see the arrow indicates that no ANSI interpretion happens for STDOUT. That's not uncommon, DOS did not interpret ANSI sequences by default, and neither did Windows. Both never needed to do so, because both DOS and Windows never ran natively on terminals. In DOS, you needed to load ANSI.SYS or similar, Windows needs some other changes, and things have changed again in some Windows 10 update. See https://stackoverflow.com/questions/16755142/ and pages linked from there, and https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences for Windows 10 changes.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
[reply]
      I followed the links in your post reading everything carefully. A comment to one of the StackOverflow questions had an easy solution to the problem, as follows (in case anyone else runs into this):
      Add the following value of type DWORD to the registry at the indicated key:

      HKEY_CURRENT_USER\Console
  VirtualTerminal = 1

      [download]
      Problem solved and thank you! I spent several hours trying to figure this one out...
[reply]
[d/l]
[select]
Re: A Win32 new Perl build dilemma
by choroba (Cardinal) on Nov 29, 2021 at 09:33 UTC
    > v5.35.6

    Are you aware that odd minor versions are development versions? The latest stable Perl version is 5.34.0.

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]
      I am aware and the issue is present in version 5.34.0, as I pointed out in my original question.
[reply]
Re: A Win32 new Perl build dilemma
by Corion (Patriarch) on Nov 29, 2021 at 07:25 UTC

    I don't use Cygwin, but I guess that all Cygwin programs run their output through an ANSI escape interpreter, while the Win32 Perl does not.

    This would explain why you see the ANSI escapes interpreted with Cygwin Perl but raw with the Win32 Perl.

    You can "fix" this by loading the module Win32::Console::ANSI in your program, which interprets the output as ANSI and instructs the Windows console to render it properly. Of course that means you should only load that module on "real" Win32 Perl, not Cygwin Perl.

    use if $^O eq 'MSWin32', 'Win32::Console::ANSI';

    [download]
[reply]
[d/l]
      Win32::Console::ANSI did not resolve the issue:
      C:\>perl -e "use warnings; use strict; use Win32::Console::ANSI; print ""\e[1;34mThis text is bold blue.\e[0m\n""; "
←[1;34mThis text is bold blue.←[0m
[reply]
Re: A Win32 new Perl build dilemma
by aitap (Curate) on Nov 29, 2021 at 07:20 UTC
    Try using WinAPIOverride to see if there's any difference in WinAPI calls between the old and the new Perl? If there isn't any, it must be somewhere else (flags in the EXE itself set by the linker? something fishy in the manifest?). If you do find something, it could be traced to the CRT and Perl's use of it.
[reply]
      Turned out to be a need for a registry change, as mentioned in my reply to the first post...
[reply]

Back to Seekers of Perl Wisdom