in reply to How can you sysread an entire file?

sysread(DF, $rec, -s DF); will save you from specifying an arbitrary number.

foreach(split("\n",$rec)){push(@array,"$_\n";)} causes three copies of the file to be in memory at once. ($rec, foreach's list and @array).

push(@array, $1) while $rec =~ /(.*?\n)/g; will save you from having an third copy of the file in memory, and will save you from re-adding the "\n". Caveat: It will skip the last line if it isn't terminated by a "\n".

Unfortunately, splitting on "\n" is not portable. Keep in mind that sysread has a bug causing it to never translate CRLF to LF in Windows.

Is push(@array, $_) while <DF>; really that much slower?

Any why not use undef $rec; instead of $rec = '';? I prefer to just use curlies, though: my @array; { my $rec; ... }

Replies are listed 'Best First'.
Re^2: How can you sysread an entire file?
by davidrw (Prior) on Jan 12, 2006 at 20:24 UTC
    Is push(@array, $_) while <DF>; really that much slower?
    Why not just @array = <DF> or push @array, <DF> ?
    Hmm.. as i was about to hit 'Create' i think i answered my own question -- those two make a copy of the whole <DF> array first, right? where your push/while avoids it ...


    Also, to OP's first part, you can use Perl Idioms Explained - my $string = do { local $/; <FILEHANDLE> }; to slurp the whole file ...
[reply]
[d/l]
[select]
      Unfortunately I don't think I can use that "idioms" as I really do need to use sysread... (Due to IO operations being counted by ISP.)
[reply]

      my $string = do { local $/; <FILEHANDLE> };
      looks nicer than
      my $string; { local $/; $string = <FILEHANDLE> }
      but takes twice as much memory. That's probably not that wise when dealing with entire files.

[reply]
[d/l]
[select]

        I think I made an interesting discovery. As you suggest above, if I do this it only makes one copy of the file in memory: 

        P:\test>perl -le"
my $s; do{ local $/; $s=<>}; 
print length $s; 
system qq[tasklist /fi \"pid eq $$\" ]" 1000000.dat

11000001

Image Name                   PID Session Name     Session#    Mem Usag
+e
========================= ====== ================ ======== ===========
+=
perl.exe                     604                         0     12,860 
+K

        [download]

        And if I do it this way it uses two copies: 

        P:\test>perl -le"
my $s=do{ local $/; <>}; 
print length $s; 
system qq[tasklist /fi \"pid eq $$\" ]" 1000000.dat

11000001

Image Name                   PID Session Name     Session#    Mem Usag
+e
========================= ====== ================ ======== ===========
+=
perl.exe                    1372                         0     23,624 
+K

        [download]

        But what fooled me, as I was using my perl shell, is that if you eval the statement, only one copy is made: 

        P:\test>perl -le"
my $s = eval q[do{ local $/; <>}]; 
print length $s; 
system qq[tasklist /fi \"pid eq $$\" ]" 1000000.dat

11000001

Image Name                   PID Session Name     Session#    Mem Usag
+e
========================= ====== ================ ======== ===========
+=
perl.exe                    1764                         0     12,880 
+K

        [download]

        And I can't quite figure out why that should be so?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
        takes twice as much memory.

        Do you have some evidence of that as it appears to make no difference at all on my system?

        Update: I take that back. It does double the memory usage.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re^2: How can you sysread an entire file?
by NeilF (Sexton) on Jan 12, 2006 at 22:23 UTC
    Thanks...

    Not sure if I fully understand what you mean by "\n" is not portable? What exactly is the implication. I've run some tests and it seems to work OK on the files I've used it on (Windows XP & Unix).

    My ISP counts IO operations. Doing a "while <DF>" would mean an IO operation (I believe) for every 512 bytes of data as it is buffered in. By using sysread the whole file is read in by one IO operation. Hence me going down this approach...
[reply]
      try this under windows and linux 
      use strict;
use warnings;

open(FILE, '>', 'foo.bar') or die "Daim!";
print FILE "\n";
close FILE;

      [download]
      under windows, the file will have a size of two bytes (\r\n)- under linux just 1 (\n) comapre it to 
      use strict;
use warnings;

open(FILE, '>', 'foo.bar') or die "Daim!";
binmode FILE;
print FILE "\n";
close FILE;

      [download]
[reply]
[d/l]
[select]
        I'm aware of that... All my files on Windows & Unix follow that rule... What's the problem?
[reply]

In Section Seekers of Perl Wisdom