open(my $fh, '<', $filename);
 binmode($fh);
[download]

If you want to read raw bytes from a filehandle (for example to look for a BOM) and _then_ add an encoding layer then you need to ensure that there isn't a :crlf layer hanging around to begin with which the above has (it may not be enabled, but it's still there *). This is because of :crlf not working nicely with encodings eg. it breaks tell().

So, the method i use is:

open(my $fh, "<:raw:perlio", $file);

# read first few bytes for a BOM or ASCII in utf(16|32)
#  to determine encoding

binmode $fh, ":encoding($encoding)"
[download]

open $fh, "<", $^X or die;
binmode $fh;
print "1: ", join(' ', PerlIO::get_layers($fh, details => 1)), "\n";
close $fh;

open $fh, "<:raw:perlio", $^X or die;
print "2: ", join(' ', PerlIO::get_layers($fh, details => 1)), "\n";
close $fh;
[download]

1: unix  4195328 crlf  4195328
2: unix  4195328 perlio  4195328
[download]

Thanks for your advice !

It's interesting that binmode() only disables the :crlf layer but doesn't remove it. (Just discovered, that's actually documented in PerlIO.)

Does it still interfere with tell() even when it's disabled ? (perlport says: "If you use binmode on a file, however, you can usually seek and tell with arbitrary values in safety.")

I'm just curious, what happens if you try these on Windows:

open $fh, "<", $^X or die;
binmode $fh, ":raw";        # perldoc says: identical to 'binmode $fh'
print "3: ", join(' ', PerlIO::get_layers($fh, details => 1)), "\n";
close $fh;

open $fh, "<", $^X or die;
binmode $fh, ":raw:perlio"; # seems not useful (see below)
print "4: ", join(' ', PerlIO::get_layers($fh, details => 1)), "\n";
close $fh;
[download]

On UNIX that gives:

3: unix  4195328 perlio  4195328
4: unix  4195328 perlio  4195328 perlio  4195328
[download]

---

I know this discussion should probably go into a Perl develompent group but let me just explain where I'm having problems to understand things here:

If one has to use open($fh, '<:raw:perlio', $file) to get a clean and buffered binary stream this unconditionally forces a :perlio layer onto the stack, which in turn inserts a :unix layer below it.

Now, what about MacOS where Perl uses :stdio by default ? (I know far too little about MacOS but it must be faster than :unix:perlio there.) Also if someone wants to change the default layers via the PERLIO environment variable (for whatever reason), it would be blithely ignored on those streams.

> Does it (a :crlf layer) still interfere with tell() even when it's disabled ?

No. But if you later add (binmode) a :crlf layer then it can enable it at the wrong 'level'. This is a common mistake when using utf(16|32) eg.

# wrong way to write utf16
open $fout, ">", $file or die;
binmode $fout, ":encoding(utf16le)" or die;
print $fout "abc\n123\n\n";
close $fout;
[download]

this results in CRLF being output as "0D 0A 00" bytes. Oops. What you need is:

# OK
open $fout, ">:raw:encoding(utf16le):crlf", $file or die;
# etc.
[download]

Bascially there are known bugs with mixing :encoding and :crlf layers and suggest that if you need both that you test well to ensure you're not hitting them. (Also check the Perl bug database).

Your code gave for me on windows:

3: unix  4195328 crlf  4195328
4: unix  4195328 crlf  4195328 perlio  4195328
[download]

> If one has to use open($fh, '<:raw:perlio', $file) to get a clean and buffered binary stream this unconditionally forces a :perlio layer onto the stack, which in turn inserts a :unix layer below it.

NB. my original point was about creating a raw byte stream that you *then* wanted to add :encoding layers to.

NB. the reason for the :perlio in  open $fin, "<:raw:perlio" is to give the filehandle the ability to be -B/-T -able. Consider:

C:\>perl -we "open $fin, '<:raw', $^X or die; die if -B $fin
-T and -B not implemented on filehandles at -e line 1.

C:\>perl -we "open $fin, '<:raw:perlio', $^X or die; die if -B $fin
Died at -e line 1.
[download]

If all you want *is* a binary, then either of:

open $fin1, "<", $foo or die;
binmode $fin1;

open $fin2, "<:raw", $bar or die;
[download]

are equivalent (and you can forget all my ramblings :-)