Re^2: What could cause excessive page faults?

because the map generates an anon array that is an "extra step" and that array is copied to @a.

No array is created. One million scalars are created, but the question wasn't about that snippet. It was provided as a baseline.

In your later code, you have a map within a map which is similar to a foreach within a foreach. So it is going to run like 1 million times slower.

No, it's not multiplicative like a foreach in a foreach. It's additive like a foreach after a foreach.

my @array = map A, map B, LIST;
[download]

is functionally similar to

my @list1;
for (LIST) {
    push @list1, B;
}
my @list2;
for (@list1) {
    push @list2, A;
}
my @array = @list2;
[download]

Snippet three should take about 0.21 + (0.21-0.06) = 0.56, but it's taking 31.48 due to excessive paging. Why is it paging so much?

Comment on Re^2: What could cause excessive page faults? Select or Download Code

Replies are listed 'Best First'.
Re^3: What could cause excessive page faults? by Marshall (Canon) on Mar 11, 2010 at 03:49 UTC
on my machine for this, I get: `C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{$ +_}1..1e6;say time-$t;" 12.703125` [download] which I think is similar to: my @array = map A, map B, LIST; I am actually surprised that on a large 64 bit machine running some kind of *nix, that there are any page faults at all. I mean why does the "simple" version page fault? Sorry, I don't know. Update: oh I see that this is Windows, I presume Win 7 instead of Vista? There are a bunch of versions of this OS, that might matter. more tests on my 32 bit Win XP Pro machine 2 GB memory, AS 5.10.1: C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..100e3;say time-$t;" 0.193822860717773 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..200e3;say time-$t;" 0.609375 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..300e3;say time-$t;" 1.28125 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..100e3;say time-$t;" 0.18930196762085 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..250e3;say time-$t;" 0.921875 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..500e3;say time-$t;" 3.375 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..1000e3;say time-$t;" 13.15625 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{ +$_}1..2000e3;say time-$t;" 50.140625 [download] Another "benchmark update": `C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=1..16000e3;s +ay time-$t;" 1.5 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}1..16 +000e3;say time-$t;" 5.546875 C:\Projects>perl -MTime::HiRes=time -wE"my $t=time; my @a=map{$_}map{$ +_}1..16000e3;say time-$t;" 3173.625` [download] The maps above essentially don't do anything useful at all, but this shows the exponential increase in execution time on my 32 bit Win machine. So, I don't think this is specific to 64 bit machines. Apparently map uses more memory than one would think for a "do nothing" operation and also that Perl winds up accessing this extra memory in a way that causes a lot of page faults which would indicate that Perl is not cycling through sequential memory locations. Why that is and how that works, I don't know yet. But at least I can say this happens on 32 bit machines also.	[reply] [d/l] [select]
Re^4: What could cause excessive page faults? by fullermd (Priest) on Mar 11, 2010 at 06:34 UTC
I am actually surprised that on a large 64 bit machine running some kind of nix, that there are any page faults at all. Are we sure there's actually even a problem? A page fault* is not the same as paging (to swap). You'll take page faults any time you use memory, the first time you touch a page after you allocate it (on most nix systems anyway; terms could* mean totally different things on Windows, but I don't think they do).	[reply]
Re^5: What could cause excessive page faults? by ikegami (Patriarch) on Mar 11, 2010 at 15:42 UTC
Are we sure there's actually even a problem Yes. The OP's third snippet should take about 0.21 + (0.21-0.06) = 0.56s, but it's taking 31.48s.	[reply]
Re^5: What could cause excessive page faults? by Marshall (Canon) on Mar 11, 2010 at 08:58 UTC
I ran another benchmark (above) with 16M numbers. I think that it is very clear that memory paging is happening on my Win 32 bit XP Pro system.	[reply]


Just another Perl shrine
	PerlMonks