in reply to ForkManager Running Real Slow

Not surprising really. You are starting & reaping a million threads, each of which call a sub that returns a constant and then dies. The cost of starting and reaping each thread is about 1000 times the cost of calling that sub. The more concurrent thread you run, the more competition there is for system resources (mostly memory), so the longer it takes to start and reap each thread.

For want of a better anology, its like having a million navvies try to dig a trench, whilst sharing one shovel.

Perhaps a better anology: a million miners dig a one-man wide tunnel, one shovel load at a time. The more you send into the tunnel, the harder it is for each man to get to the digface and get his one shovel full.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP PCW It is as I've been saying!(Audio until 20090817)

Replies are listed 'Best First'.
Re^2: ForkManager Running Real Slow
by SuicideJunkie (Vicar) on Sep 04, 2009 at 13:28 UTC

    In this case, I think he has four shovels available (quad core). The mosh pit mine analogy still holds however. Throw in a long, slow, creaky elevator ride for the start/reap of miner threads. Shoving a million people into the mine and bringing them back out takes far longer than it does to actually swing the four shovels a quarter million times each.

    The code should probably be refactored into four guys sharing the four shovels, with a list of places to dig next after they finish their current one, and only take the expensive elevator back up to the surface when everything is done. If there is a lot of I/O paperwork to do before swinging the shovel each time, then you could use three or four miners per shovel; most of them won't need the shovel at any one time.

[reply]

      Indeed. Of course, to have four men at the face requires the tunnel be 4 times as wide, and that slows forward progress somewhat. But, if you factor iin the reduction in the time you spend swapping people, there is a net gain. Done right it a 3x+ net gain.

      And if you have a 4-wide tunnel and 8 shovels, although only 4 can dig, you save time swapping shovels. The next shift take the previous shift shovels while the current shift get straight to digging.

      Hm. Did we just stretch an anology? :)

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Well, the task does need to be massively parallelizable, since in theory it is possible to have a million threads working simultaneously. I suppose it isn't so much a tunnel as a sprawling mine webbed with shafts. But still only 4 shovels to go around.

        • Worker = thread
        • Shovel = CPU core
        • Passing shovel between workers = CPU context switch
        • Designated dig area/volume= combined list of thread tasks, order generally unimportant
        • Elevator (requires a shovel to operate the controls) = Spawn/kill threads
        • Boss leaning on a shovel for photo ops = CPU wasted on refreshing your desktop
        I'm not sure what RAM and swapping to disk would correspond to.
[reply]

In Section Seekers of Perl Wisdom