in reply to Re: Character encoding in console in Windows
in thread Character encoding in console in Windows

Thanks for the more dumb-proof explanation, although I'm afraid it's still not dumb-proof enough for me.

If you're entering the file name by dragging and dropping, Windows should be encoding them using the console's code page, which means it's already properly encoded for use by open(). Don't do any decoding, and it should work.
Drag&drop would indeed be sufficient for me, although I'm not sure if there's any difference between typing a path in the console and having it pasted in with drag & drop. Either way, I certainly don't need @ARGV filename input at all. You say drag and drop should work with plain old open(), but this is exactly what I originally tried and it failed, hence this tread. To make things clear, here's a simplified script showing what I do. It doesn't even parse the file name for path/filename/extension, it just strips whitespace and quotes and tests if the perl script can find the file.

#!/usr/bin/perl
use strict;
use warnings;

print "Drop input file in console\n";
chomp (my $file = <STDIN>);
$file =~ s/ *["']?([^"']*)["']? *$/$1/;        # strip whitespace and 
+quotes
if (-e "$file") {print "\nFile found, everything is fine.\nFile:>$file
+<"} else {print "\nOoops, file not found.\nFile: >$file<"}
<STDIN>;

[download]

If I drop in a file called i.txt, it finds it, so far so good. If I drop in í.txt from the same folder, the script can't find it. open (FILE, "<:encoding(UTF-8)", "$file"); fails on í.txt just the same as (-e "$file"). Are you saying this should work out of the box? Or should I use one of the modules? Win32::Unicode::File? Or Win32API::File?
This command in Win32::Unicode::File seems to promise to do what I want: my $fh = Win32::Unicode::File->new($mode, $file_name); # create an instance and open the file
but it's certainly not just a plain open() like you say. I haven't installed the module so I haven't tried it.

Here's the output from your script, if I start it in the folder where í.txt is saved: 
Enter file path> í.txt
$qfn="\241.txt"
open í.txt: No such file or directory

[download]

Replies are listed 'Best First'.
Re^3: Character encoding in console in Windows
by ikegami (Patriarch) on Sep 16, 2010 at 06:41 UTC 
    >perl -MDevel::Peek -e"chomp($_=<STDIN>); Dump($_); open($fh, '<', $_)
+ or die; print <$fh>"
C:\Users\ikegami\í.txt
SV = PV(0xeaf80) at 0x2f59c8
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0x2f3788 "C:\\Users\\ikegami\\\241.txt"\0
  CUR = 22
  LEN = 80
Died at -e line 1, <STDIN> line 1.

    [download]

    Is that encoded for my ANSI code page (1252) or my OEM code page (437)? 

    >perl -MDevel::Peek -MEncode -e"Dump(encode('cp1252', chr(0xED)))
SV = PV(0x2fa570) at 0x27b210
  REFCNT = 1
  FLAGS = (TEMP,POK,pPOK)
  PV = 0x32c24f8 "\355"\0
  CUR = 1
  LEN = 8

>perl -MDevel::Peek -MEncode -e"Dump(encode('cp437', chr(0xED)))
SV = PV(0x30a570) at 0x28b210
  REFCNT = 1
  FLAGS = (TEMP,POK,pPOK)
  PV = 0x32c24f8 "\241"\0
  CUR = 1
  LEN = 8

    [download]

    So open() expects the name to be encoded using the ANSI code page, but it's coming from STDIN in the OEM code page. 

    >perl -MDevel::Peek -MEncode=from_to -e"chomp($_=<STDIN>); from_to($_,
+ 'cp437', 'cp1252'); Dump($_); open($fh, '<', $_) or die; print <$fh>
+"
C:\Users\ikegami\í.txt
SV = PV(0x159adc0) at 0x315b58
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0x344d258 "C:\\Users\\ikegami\\\355.txt"\0
  CUR = 22
  LEN = 80
ok

    [download]

    In broad strokes, the OEM code page is the encoding used by console apps, ANSI for others. Now how do you get those code pages? Good question.

[reply]
[d/l]
[select]
      Now how do you get those code pages?

      using chcp? 

      c:\test>chcp && perl -E" say `chcp`; say `chcp 437`; say`chcp`" && chc
+p
Active code page: 850
Active code page: 850

Active code page: 437

Active code page: 437

Active code page: 437

      [download]

      Alternatively, Win32::Console:

      InputCP [codepage] 

Gets or sets the input code page used by the console. Note that this d
+oesn't apply to a console object, but to the standard input console. 
+This attribute is used by the Write method. See also: OutputCP.

Example:
    $codepage = $CONSOLE->InputCP();
    $CONSOLE->InputCP(437);
    # you may want to use the non-instanciated form to avoid confuzion
+ :)
    $codepage = Win32::Console::InputCP();
    Win32::Console::InputCP(437);

      [download]
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

        What about the other? I knew how to get that one, but I didn't have time then to find how to get the ANSI code page.

        Could always use Win32::API, of course. The functions are GetACP and GetOEMCP.

[reply]
        From my brief research ( microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/chcp.mspx?mfr=true ), it looks like trying to set the console encoding with system (chcp XXX) seems like a bad idea. It doesn't even seem possible to set it to UTF-8, and setting it to anything else will probably make it impossible to use the very characters that are causing the problems in the first place, because there is no single OEM codepage that covers all the characters people may use... and I'd be reluctant to mess with the settings of other people's computers anyway (does this setting only affect the current session or is it permanent?)

        So reading the console's encoding (which depends on OS localization) and then converting the incoming text in Perl accordingly sounds better to me... but I'm just taking stabs in the dark. I can't follow half of the posts here, but I can't see working code in any of them so far.

        BTW as I said before, this is just one half of the issue.
        Even if I were to go 
        #!/usr/bin/perl
use strict;
use warnings;
use utf8;

open(FILE, "<:encoding(UTF-8)", "c:\\folder\\í.txt") or print "Oops, c
+an't open file: $!";
<STDIN>;

        [download]
        ..and save this in UTF-8, it would still fail to open the file. It seems pretty clear that I'd need to use one of the modules to ever be able to open a file with a non-ASCII name, and I can't really make sense of the documentation of the modules.

        So the step-by-step seems to be:
        1) read what the console's OEM encoding is
        2) convert filepath received via STDIN from OEM to UTF-8
        3) open the file using one of the Unicode modules

        ...or maybe I'm completely wrong.
[reply]
[d/l]
[select]
Re^3: Character encoding in console in Windows
by ikegami (Patriarch) on Sep 15, 2010 at 20:27 UTC
    You always catch me away from my Windows machine! I'll do some testing tonight.
[reply]

In Section Seekers of Perl Wisdom