Re^2: binmode(':encoding(UTF-8)') did not produce utf8 for me

Hello ikegami,

thanks very much for a fast response!

I ran the first of your scripts (without 'use utf8;') in a PowerShell console with code page 850 in Strawberry Perl 5.32.1 32-bit. Without redirection I get the same output as you.

With redirection however I get this file content:

000000 ff fe 1c 25 f1 00 1c 25 c2 00 1c 25 5d 25 1c 25
000010 e4 00 1c 25 fb 00 1c 25 a3 00 1c 25 92 01 0d 00
000020 0a 00
[download]

This is windows 10 version 21H2 (Build 19044.3086).

That made me curious, and I repeated the run in a plain CMD.EXE shell. And surprise! I then get the same output as you did.

So something in PowerShell disturbs the output during redirection, it seems. That nasty behavior took me by surprise. It looks like that is where I get the unwanted extra UTF-16 LE BOM conversion from...

Another thing I noticed with PowerShell was that perl "-Mv5.14" -ne"say sprintf '%vX', $_" a did only produce two empty lines.

Thanks very much again for helping me out!

Comment on Re^2: binmode(':encoding(UTF-8)') did not produce utf8 for me Select or Download Code

Replies are listed 'Best First'.
Re^3: binmode(':encoding(UTF-8)') did not produce utf8 for me by ikegami (Patriarch) on Jul 05, 2023 at 03:29 UTC
That has nothing to do with Perl. That's PowerShell doing that. You can control the output as follows: `script \| Out-File -Encoding UTF-8 file.out` [download] The above outputs a BOM. The following doesn't, but requires PS6+ `script \| Out-File -Encoding UTF8NoBOM file.out` [download] That said, `-Encoding UTF8NoBOM` is the default with PS6+, so all you need is the following if you're using PS6+: `script >file.out` [download] So the issue is really that you are using an outdated PowerShell. References: Changing PowerShell's default output encoding to UTF-8. Using PowerShell to write a file in UTF-8 without the BOM.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: binmode(':encoding(UTF-8)') did not produce utf8 for me
by ikegami (Patriarch) on Jul 05, 2023 at 03:29 UTC

That has nothing to do with Perl. That's PowerShell doing that. You can control the output as follows:

script | Out-File -Encoding UTF-8 file.out
[download]

The above outputs a BOM. The following doesn't, but requires PS6+

script | Out-File -Encoding UTF8NoBOM file.out
[download]

That said, -Encoding UTF8NoBOM is the default with PS6+, so all you need is the following if you're using PS6+:

script >file.out
[download]

So the issue is really that you are using an outdated PowerShell.

References:

[reply]
[d/l]
[select]