Perl, DOS and encodings

siberia-man has asked for the wisdom of the Perl Monks concerning the following question:

Hello dear Monks,

I came with the issue I have encountered recently.

I have the perl script git-md-toc helping me to generate Table of Content (TOC) from a markdown file and embed it into the original file.

It worked fine with the Latin charset. Later I found it doesn't work with other encodings. I extedned it to support other encodings by specifying a particular encoding via an additional command line option. It works fine as well (I tested it under Cygwin). However it fails under DOS session, if there is need to add a title of TOC to the file written with non-Latin charset/encoding.

For example, there is test file in UTF8 having some Cyrillic text. I need to update it adding TOC with the title in Russian.

This command in bash works fine (Perl 5.30 shipped with Cygwin):

git-md-toc -ut "some-text-in-russian" -Tutf8 "utf8-cyrillic.md"
[download]

But it fails in DOS sessions -- the title is being added in wrong encoding. To resolve the issue I have to use one more option (standalone StrawberryPerl 5.30):

git-md-toc --title-transcode=cp1251 -ut "some-text-in-russian" -Tutf8 
+"utf8-cyrillic.md"
[download]

The thing confusing me is that the default DOS code page is 866 and the encoding for the title I have to specified is 1251.

My questions are:

Is this something specific for DOS, Perl or combination of both?
How does the script work in other systems (windows, linux especially with encodings unlike of mine)?

Comment on Perl, DOS and encodings Select or Download Code

Replies are listed 'Best First'.
Re: Perl, DOS and encodings by haj (Vicar) on Apr 29, 2020 at 19:37 UTC
This is indeed related to weirdness of the `cmd.exe` command box. You can control the output encoding of Perl programs with the `chcp` command, but this does not affect the encoding of Perl's `@ARGV`. I found this superuser.com anwser helpful to find out the default encoding for my machine, and it is this encoding which is applied to parameters which you pass to Perl programs, regardless of your `chcp` settings. So, most probably, your Windows system is using the cyrillic default encoding of codepage 1251 for input - but defaults to codepage 866 for output. Cygwin is another story, of course. Contemporary Unix/Linux terminals are using UTF-8 as default encoding, and this is applied when you pass data from bash (the Cygwin shell) to your Perl program.	[reply]
Re^2: Perl, DOS and encodings by siberia-man (Friar) on Apr 29, 2020 at 20:02 UTC
Thank you for your response. I've just tested the suggestion from the superuser.com answer. To be honest, without your explanation that answer doesn't give much clues. Simply compare it. As I have already said, the code page defaults to `866` (or IBM CP866, the old code page since MSDOS 4.01). `BodyName = koi8-r` is another code page 20866. How does it indeed work -- I don't know, cmd.exe is definitely painful. `C:\>chcp Active code page: 866 C:\>powershell -c "[System.Text.Encoding]::Default" IsSingleByte : True BodyName : koi8-r EncodingName : Cyrillic (Windows) HeaderName : windows-1251 WebName : windows-1251 WindowsCodePage : 1251 IsBrowserDisplay : True IsBrowserSave : True IsMailNewsDisplay : True IsMailNewsSave : True EncoderFallback : System.Text.InternalEncoderBestFitFallback DecoderFallback : System.Text.InternalDecoderBestFitFallback IsReadOnly : True CodePage : 1251` [download] I tested the command from my opening post with different codes pages, setting it to `1251` or `65001` (utf-8). The only correct encoding for Cyrillic text in CLI is `1251`. The default encoding in Cygwin is `en_US.UTF-8`. Updated: I tested the script invoking it from the shell/batch script. It works correctly, if the title's encoding corresponds the encoding of the shell script. The code page 1251 only has to be specified in the batch script, independently of the encoding of the batch script itself.	[reply] [d/l] [select]
Re^3: Perl, DOS and encodings by haj (Vicar) on Apr 29, 2020 at 22:10 UTC
The relevant information is the `WindowsCodePage` entry. This is the encoding which is used by `cmd.exe` to pass cyrillic characters from your terminal input to your Perl program, and you can not change it using `chcp`. According to my experiments, which may be totally bogus, things get even more interesting if you write your command, including command line parameters with cyrillic characters, into a `.bat` file and execute that. In that case, the `chcp` setting will be used to decode the batch file - but still the Perl program will receive its `@ARGV` in the `WindowsCodePage` encoding. So, if your batch file is UTF-8 encoded, you need to `chcp 65001` and use `--title-transcode=cp1251` if you pass the title as a command line parameter.	[reply]
Re^4: Perl, DOS and encodings by siberia-man (Friar) on Apr 30, 2020 at 07:24 UTC
Re: Perl, DOS and encodings by Anonymous Monk on Apr 29, 2020 at 23:43 UTC
`This command in bash works fine (Perl 5.30 shipped with Cygwin): git-md-toc -ut "some-text-in-russian" -Tutf8 "utf8-cyrillic.md" But it fails in DOS sessions -- the title is being added in wrong enco +ding. To resolve the issue I have to use one more option (standalone +StrawberryPerl 5.30):` [download] Its simple, avoid "dos sessions", use perl `#!/usr/bin/perl -- use strict; use warnings; use utf8; system $^X, q{-e}, q{ use strict; use warnings; use Encode; use Data::Dump; my @argv = map{ decode('UTF-8', $_, Encode::FB_CROAK) } @ARGV; dd\@argv; }, qq{\x{43A}\x{435}\x{431}\x{430}\x{431}}, # kebab # copy from html below cause perlmonks doesnt utf, so browser converts + html entities 'кебаб', ;` [download] As you can see from dd output below, кебаб is кебаб Since perlmonks doesnt utf8 the browser converts to entities `кебаб` `[ "\x{43A}\x{435}\x{431}\x{430}\x{431}", "\x{43A}\x{435}\x{431}\x{430}\x{431}", ]` [download] Why avoid dos sessions, at least unicode ones? Because its complicated. At one time I had a handle on this knowledge but long before that decided its simpler/more portable to avoid unicode in filenames and shells How Command Line Parameters Are Parsed by David Deley Quotes, Escape Characters, Delimiters - Windows CMD - SS64.com Re: Non asci character and system call Re^4: How can I get a Unicode @ARGV? How can I get a Unicode @ARGV? windows - How to make Unicode charset in cmd.exe by default? - Stack Overflow Re^2: Windows 7: fork in perl script spawns new cmd window Re: Parsing Windows CommandLine from Perl Re: eval NoOp on cmd.exe line expr Discipulus library chcp can't do everything - Sorting it all Out - Site Home - MSDN Blogs http://stackoverflow.com/questions/10764920/utf-16-on-cmd-exe http://stackoverflow.com/questions/2182568/batch-script-is-not-executed-if-chcp-was-called ConEmu - Handy Console Window download Re: Perl in Windows - STDOUT to file Behavior difference by run context Re^5: Out of date over <> and 5.10 (reproduce) redirection problem Win7, ASSOCiation , and @ARGV Upside Down Text Revised Windows console mangles UTF8 output Re: Unicode again, in Win7 cmd (chcp 65001) chcp can't do everything chcp can't do everything - Sorting it all Out - Site Home - MSDN Blogs https://illegalargumentexception.blogspot.com/2009/04/i18n-unicode-at-windows-command-prompt.html	[reply] [d/l] [select]
Re^2: Perl, DOS and encodings by siberia-man (Friar) on Apr 30, 2020 at 07:28 UTC
Thank you for sharing a lot of links. I will read them later.	[reply]