Re^5: memory use array vs ref to array

You don't need the do if you don't use the value of the last expression.

True. But it's habitual.

Moreover, I tried both ways on a 2GB file and didn't notice any difference in memory consumption:

I'm not familiar with the ins and outs of memory measurement on *nix; but I can demonstrate the difference on Windows.

C:\test>p1
[0]{} Perl> print mem;;
9,432 K

[]{} Perl> open I, '<', 'test.dat'; my $s; do{ local $/; $s = <I> }; p
+rint mem;;
997,784 K

[]{} Perl> Terminating on signal SIGINT(2)

C:\test>p1
[0]{} Perl> print mem;;
9,440 K

[]{} Perl> open I, '<', 'test.dat'; my $s = do{ local $/; <I> }; print
+ mem;;
1,986,060 K

[]{} Perl> Terminating on signal SIGINT(2)
[download]

As you can see, in the latter case, the memory assigned to the process is double: ( 997784 - 9432 ) * 2 + 9440 = 1986146; almost exactly the 1,986,060 K measured in the latter case.

The reason is that in the later case, the data is read into an internal mortal temporary scalar; and then copied from there to the named lexical, before the memory attached to the temp is freed. As the allocation is greater than (from memory) 1MB, (on windows at least) such huge allocations are allocated directly from the OS's virtual memory rather than from the process' heap; and then get released directly back to OS.

Which is usually a good thing, but it still means you need to have double the memory available for a short while, and if you are close to the limits, that can blow the process.

Is it possible that large allocations are also freed back to the OS on *nix, and you are measuring after it has been freed?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^5: memory use array vs ref to array Download Code

Replies are listed 'Best First'.
Re^6: memory use array vs ref to array by choroba (Cardinal) on Sep 17, 2016 at 22:00 UTC
> Is it possible that large allocations are also freed back to the OS on nix, and you are measuring after it has been freed?* I'm not sure where this `mem` comes from. I tried with Memory::Usage, measuring the consumption with top running with 0.001s delay (was probably more in practice): `#!/usr/bin/perl use warnings; use strict; use feature qw{ say }; use Memory::Usage; my $file = shift; open my $FH, '<', $file or die $!; my $string; my $mu = 'Memory::Usage'->new; $mu->record('start'); { local $/; $string = <$FH> }; $mu->record('after do'); undef $string; seek $FH, 0, 0; $string = do { local $/; <$FH> }; $mu->record('after no do'); $mu->dump;` [download] Output: `time vsz ( diff) rss ( diff) shared ( diff) code ( diff) + data ( diff) 0 17504 ( 17504) 3916 ( 3916) 3328 ( 3328) 1644 ( 1644) + 1060 ( 1060) start 0 1970632 ( 1953128) 1957112 ( 1953196) 3520 ( 192) 1644 +( 0) 1954188 ( 1953128) after do 1 1970632 ( 0) 1957248 ( 136) 3520 ( 0) 1644 ( + 0) 1954188 ( 0) after no do` [download] Top output: Read more... (26 kB) ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^7: memory use array vs ref to array by BrowserUk (Patriarch) on Sep 17, 2016 at 22:24 UTC
It looks like you've got your "after do" and "after no do" labels the wrong wayaround; but I agree, it looks like there is no difference on your version of Perl on *nix. I was running under my default perl5.10; so I re-ran the tests under the latest version I have installed, 5.22 and I see the same results: `C:\test>\Perl22\bin\p1.pl [0]{} Perl> print mem;; 9,404 K []{} Perl> open I, '<', 'test.dat'; my $s; do{ local $/; $s = <I> }; p +rint mem;; 997,760 K []{} Perl> Terminating on signal SIGINT(2) C:\test>\Perl22\bin\p1.pl [0]{} Perl> print mem;; 9,392 K []{} Perl> open I, '<', 'test.dat'; my $s = do{ local $/; <I> }; print + mem;; 1,986,016 K []{} Perl> Terminating on signal SIGINT(2)` [download] So maybe it's 'a windows thing'. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^7: memory use array vs ref to array by BrowserUk (Patriarch) on Sep 18, 2016 at 01:34 UTC
I'm not sure where this mem comes from. BTW. I missed/ignored that at first reading. mem() is a function defined in my REPL: sub mem{ `tasklist /nh /fi "PID eq $$"` =~ m[(\S+ K)$] } [download] And tasklist is a system utility, which with the above parameters produces: C:\test>\Perl22\bin\p1 [0]{} Perl> print `tasklist /nh /fi "PID eq $$"`;; perl.exe 2524 Console 1 +9,384 K [download] And without the /n{o}h{eader} switch gives: [0]{} Perl> print `tasklist /fi "PID eq $$"`;; Image Name PID Session Name Session# Me +m Usage ========================= ======== ================ =========== ===== +======= perl.exe 2524 Console 1 +9,428 K [download] With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]
Re^7: memory use array vs ref to array by Anonymous Monk on Sep 18, 2016 at 10:17 UTC
Please try `sub psMem { my( $pid ) = @_; my @lines = qx{$GOTPS -o vsize,rss -p $pid }; chomp @lines; foreach my $line (@lines) { my ( $vsize, $rss) = split /\s+/, $line; return "VM: $vsize RSS: $rss "; } return; }` [download]	[reply] [d/l]
Re^8: memory use array vs ref to array by choroba (Cardinal) on Sep 18, 2016 at 12:26 UTC
I don't know what $GOTPS is. But I've run `#!/usr/bin/perl use warnings; use strict; use feature qw{ say }; say $$; sleep 10; my $file = shift; open my $FH, '<', $file or die $!; my $string; { local $/; $string = <$FH> }; say length $string; sleep 1; undef $string; seek $FH, 0, 0; $string = do { local $/; <$FH> }; say length $string;` [download] I've used the PID reported by the script in another terminal and ran: `while : ; do ps -o vsize,rss -p $PID \| grep '[0-9]' ; done` [download] And got the following output: Read more... (6 kB) ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^9: memory use array vs ref to array by Anonymous Monk on Sep 18, 2016 at 22:00 UTC
Re^10: memory use array vs ref to array by choroba (Cardinal) on Sep 19, 2016 at 07:55 UTC