Re: Thread::Queue memory issue with nested maps but not foreach loops...

I would like to know why the nested maps produces the behavior?

First. Take threads and Thread::Queue out of the equation. They are innocent bystanders in the issue.

Using nested maps, this require 49MB and 11.4 seconds of cpu time to complete:

C:\test>perl -E"$c=0; map map( ++$c, 1..1e3 ), 1..1e3; say 'check mem'
+;<>"
check mem
49 MB
[download]

Whereas, this using nested for loops requires just 2 1/2 MB and 0.014 seconds of cpu:

C:\test>perl -E"$c=0; for( 1..1e3 ) { ++$c for 1..1e3 }; say 'check me
+m';<>"
check mem
2.5MB
[download]

For why,

map operates on lists -- so 1 .. 1e3 builds a 1000 item list on "the stack" -- and nesting them means that 1000 lists of 1000 items need to be built.
for will(*) process 1 .. 1e3 as an iterator, grabbing one value at a time as it needs it.

(*)for will also build a list in some circumstances, but far less frequently.

It pays to know (some of) the internal details of your language.

(As an aside, filling Thread::Queues will huge numbers of items costs big in terms of memory and runtime. Better to limit how much you push into them at one time).

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Comment on Re: Thread::Queue memory issue with nested maps but not foreach loops... Select or Download Code

Replies are listed 'Best First'.
Re^2: Thread::Queue memory issue with nested maps but not foreach loops... by ikegami (Patriarch) on Mar 03, 2012 at 08:41 UTC
`for (EXPR .. EXPR)` [download] is an iterator, but `for (CONSTANT .. CONSTANT)` [download] is actually `my @anon; BEGIN { @anon = (CONSTANT..CONSTANT); } for (@anon)` [download] Now, `for (@anon)` is treated more efficiently than generic lists, but I don't remember how so exactly.	[reply] [d/l] [select]
Re^3: Thread::Queue memory issue with nested maps but not foreach loops... by BrowserUk (Patriarch) on Mar 03, 2012 at 09:13 UTC
Sorry, but that simply cannot be true. At least not for large ranges: C:\test>perl -MTime::HiRes=time -E"$t=time; ++$c for 1..1e6; say time- +$t; <>" 0.0741260051727295 5.2 MB C:\test>perl -MTime::HiRes=time -E"$e=1e6;$t=time; ++$c for 1..$e; say + time-$t; <>" 0.0663371086120605 5.3 MB C:\test>perl -MTime::HiRes=time -E"$t=time; ++$c for 1..1e7; say time- +$t; <>" 0.635999917984009 5.2 MB C:\test>perl -MTime::HiRes=time -E"$e=1e7;$t=time; ++$c for 1..$e; say + time-$t; <>" 0.645999908447266 5.3 MB C:\test>perl -MTime::HiRes=time -E"$t=time; ++$c for 1..1e8; say time- +$t; <>" 6.22199988365173 5.2 MB C:\test>perl -MTime::HiRes=time -E"$e=1e8;$t=time; ++$c for 1..$e; say + time-$t; <>" 6.46099996566772 5.3 MB C:\test>perl -MTime::HiRes=time -E"$t=time; ++$c for 1..1e9; say time- +$t; <>" 61.9520001411438 5.2MB C:\test>perl -MTime::HiRes=time -E"$e=1e9;$t=time; ++$c for 1..$e; say + time-$t; <>" 64.4389998912811 5.3 MB [download] There isn't any evidence for it at small range sizes: `C:\test>perl -MTime::HiRes=time -E"$t=time; ++$c for 1..1e2; say time- +$t; <>" 2.09808349609375e-005 5.2 MB C:\test>perl -MTime::HiRes=time -E"$e=1e2;$t=time; ++$c for 1..$e; say + time-$t; <>" 1.9073486328125e-005 5.3 MB` [download] With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l] [select]
Re^4: Thread::Queue memory issue with nested maps but not foreach loops... by ikegami (Patriarch) on Mar 05, 2012 at 06:55 UTC
I stand corrected. I got two things mingled. It only flattens a constant range at compile-time when it would otherwise be flattened at run-time. (e.g. `my @a = (1..10_000);`)	[reply] [d/l]
Re^2: Thread::Queue memory issue with nested maps but not foreach loops... by Anonymous Monk on Mar 03, 2012 at 05:15 UTC
Everything in your post makes sense to me; however, I have a single threaded version of the program that uses the same amount of memory using both the nested maps and the foreach loop. Also, aren't newer versions of perl optimized such that when map is used in a void context, the extra work that differentiates it from foreach and from for optimized away? (Source: http://www.perlmonks.org/index.pl?node_id=296742) The threaded version and the single threaded version are basically identical except instead of loading a queue, I call &doWork on the string. If the Thread::Queue was not part of the problem, shouldn't nested map use more memory than the nested foreach loops? From my testing, the outrageous use of memory only occurs when I am using Threads and Thread::Queue, otherwise the two perform similarly. Thankyou	[reply]
Re^3: Thread::Queue memory issue with nested maps but not foreach loops... by BrowserUk (Patriarch) on Mar 03, 2012 at 05:20 UTC
Also, aren't newer versions of perl optimized such that when map is used in a void context, the extra work that differentiates it from foreach and from for optimized away? (Source: http://www.perlmonks.org/index.pl?node_id=296742) The optimisation relating to map in a void context is that map doesn't build the return list (what would be returned from the map) when it detects it is called in a void context. It doesn't (I think "can't", but I not sure of that), stop the input list being built. From my testing, the outrageous use of memory only occurs when I am using Threads and Thread::Queue, otherwise the two perform similarly. Post -- or send me via the mail ID on my home page -- the real code. Then we'll see. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^4: Thread::Queue memory issue with nested maps but not foreach loops... by ikegami (Patriarch) on Mar 03, 2012 at 08:40 UTC
It doesn't (I think "can't", but I not sure of that), stop the input list being built. It could be done. The opcode checker can transform the opcode tree when it detects a pattern that can be optimised. It can even be implemented in a module outside of core.	[reply]
Re^5: Thread::Queue memory issue with nested maps but not foreach loops... by BrowserUk (Patriarch) on Mar 03, 2012 at 09:41 UTC