PhilHibbs has asked for the wisdom of the Perl Monks concerning the following question:

I have a script that reads STDIN by setting $/ to \16 and uses while (<>) to read 16 bytes at a time, then outputs the hex values of each byte. On MS-Dos based systems, it needs to read STDIN in binmode in order to represent the line-ends correctly. This works if the script is invoked with the < operator to feed the file in, or fed from a pipe, but if the file name is passed as a parameter, the (<>) magic does not respect the binmode setting.

This script is in a .cmd file so it can be invoked directly from the Command Prompt. Ignore everything up to the use strict; as this is a standard Perl/Batch polyglot. 

@rem = '--*-Perl-*--
@echo off
perl "%~dpnx0" %1 %2 %3 %4 %5 %6 %7 %8 %9
goto endofperl
@rem ';
#!perl
#line 8
use strict;
use warnings;

binmode(STDIN);
$/ = \16;
my $line = '';
my $len = 0;
my $offset=0;
my $offhex='';
my $ord=0;
my $hex='';
my $char='';
print <<HEAD;
Offset    00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  0123456789A
+BCDEF
--------  -----------------------------------------------  -----------
+-----
HEAD
format STDOUT =
@<<<<<<<< @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<
+<<<<<
$offhex,  $hex,                                            $char,
.
while (<>) {
  $len = length;
  $offhex = sprintf('%08X', $offset);
  $hex = '';
  $char = '';
  foreach (split (//)){
    $ord = ord($_);
    $hex  .= sprintf('%02X ', $ord);
    $char .= ($ord > 13 or $ord < 7) ? $_ : ".";
  }
  write;
  $offset += $len;
}
__END__
:endofperl

[download]

Update: If I set the environment variable PERLIO=:raw as per ysth's suggestion, it works for files passed as parameters. This breaks redirected input, though! So, my script just tests %1 and only sets the PERLIO variable if there is a parameter.

Replies are listed 'Best First'.
Re: binmode(STDIN)
by tye (Sage) on Feb 08, 2005 at 18:08 UTC

    Long ago I noticed that open.pm didn't work for this case so I started patching Perl to fix the problem but ran into p5p's insistence that <> must be able to delete files or e-mail /etc/password or whatever else makes absolutely no sense for a file-reading operator to do.

    So you might be able to misuse this misfeature of <> with something like: 

    BEGIN { $_= "<:raw $_" for @ARGV }

    [download]

    (untested)

    - tye        

[reply]
[d/l]
[select]
      ! But that will interfere with deleting files, emailing /etc/passwd, or whatever.
[reply]
Re: binmode(STDIN)
by nobull (Friar) on Feb 08, 2005 at 18:13 UTC
    I have a script that reads STDIN by setting $/ to \16 and uses while (<>)
    while (<>) does not read from STDIN, it reads from the special magic file handle ARGV. If if there are no arguments passed into @ARGV then ARGV is STDIN.

    I seem to recall reading somewhere that the only way to get the ARGV magic to open the files listing in @ARGV binary mode is to use open and that this is only available in 5.8 onwards.

[reply]
[d/l]
[select]
      What does use open mean in this context? Should I just put a line that says "use open;" in my code? I've never heard of the open module.

      PhilHibbs runs 'perldoc open' and reads it

      Update: I tried putting use open ':bytes'; in, but that fails:
      Unknown PerlIO layer class ':bytes' at C:\bin\hexdump.cmd line 10

      Update: I tried putting use open ':utf8'; in, but that makes no difference, it still only outputs 0A for a line-end if the file name is a parameter, but 0D 0A if it's piped or redirected in.

      Update: It's just clicked - this is what tye is talking about in his reply above.

[reply]
[d/l]
[select]
Re: binmode(STDIN)
by ysth (Canon) on Feb 09, 2005 at 03:15 UTC
    Try setting PERLIO=:raw before starting perl, but that may interfere with anything that doesn't want binmode.
[reply]
      That works! Yayhay! I can do it safely, too, because it's a Windows Command Prompt script that runs perl with itself as input, so it can set PERLIO=:raw and then un-set it afterwards. [ysth]++!
[reply]
      Hm. Now I get this:

      utf8 "\xFF" does not map to Unicode at C:\bin\hexdump.cmd line 40, <> chunk 2.

      Any suggestions?

      Update: This isn't caused by your sggestion, it happens whenever I pass the filename as a parameter rather than redirecting, though. Looks like another symptom of the same original problem.

      Update: I have fixed it by removing the use open ':utf8'; that I had added previously!

      Update: ARGH! Now it doesn't work for redirected input! That's OK, though, I can test for parameters before calling perl.

[reply]
[d/l]
Re: binmode(STDIN)
by BrowserUk (Patriarch) on Feb 08, 2005 at 19:24 UTC

    Update:This code doesn't work properly. Author's request:Please downvote. 

    perl -ple"INIT{binmode(*ARGV,':raw');$/=\16}" 
-e"$_=join $\,join' ',unpack'(A8)*',unpack'H*',$_" perldb.ini

24454e56 7b504552 4c44425f 4f505453
7d3d224e 6f6e5374 6f702066 72616d65
3d32223b 0a

    [download]
    Examine what is said, not who speaks.
    Silence betokens consent.
    Love the truth but pardon error.
[reply]
[d/l]

      If that is a normal DOS text file and your trick worked, then the output should include "0d0a" not just "0a".

      Perhaps you should double check that the binmode took effect and post your evidence.

      You pretty much have to get the perl excutable to do the binmode for you since <> doesn't open the file until it also reads the first line from the file, so Perl scripting code has no way to insert a call to binmode between the implicit open and the reading.

      This is a perfect use for open.pm, except it doesn't work with <> (unless that has been fixed recently).

      - tye        

[reply]
[d/l]
[select]

        Hmmm, This was the original test I did: 

        [19:50:37.28] P:\test>perl -ple"INIT{binmode(*ARGV,':raw');$/=\64}" -e
+"$_=join $\,join' ',unpack'(A8)*',unpack'H*',$_" 2.log
313a2063 6f6d3a31 3a203e20 6c6f6766 696c653a 322e6c6f 670a0d0a 313a206
+3 6f6d3a31 3a203e20 6c6f6766 696c653a 322e6c6f 670a0d0a 313a2063 6f6d
+3a31
3a203e20 6c6f6766 696c653a 322e6c6f 670a0d0a 313a2063 6f6d3a31 3a203e2
+0 6c6f6766 696c653a 322e6c6f 670a0d0a 313a2063 6f6d3a31 3a203e20 6c6f
+6766
696c653a 322e6c6f 670a0d0a 313a2063 6f6d3a31 3a203e20 6c6f6766 696c653
+a 322e6c6f 670a0d0a 313a2063 6f6d3a31 3a203e20 6c6f6766 696c653a 322e
+6c6f
670a0d0a

[ 3:00:14.96] P:\test>

        [download]

        Which certainly does have (several) of the 0d0a sequences.

        But looking more closely, it's actually '0a0d0a' which my eye's did not discern, I should have used a larger font.

        I admit it. I saw what I wanted to see, and then faffed around with getting the formatting right.

        Sorry.

        Examine what is said, not who speaks.
        Silence betokens consent.
        Love the truth but pardon error.
[reply]
[d/l]

Back to Seekers of Perl Wisdom