in reply to Re^6: Strange memory leak using just threads
in thread Strange memory leak using just threads

Hence the existence of forks.pm. It has the improved efficiency of creating a new "thread" because it uses fork() to do that (clearly not something to use with a native-Windows Perl, of course). (It offers the same interface as threads.pm.)

I haven't used forks.pm myself. But I have worked with POE and I recently evaluated work done by some very smart co-workers who were using similar systems. People who work with something like POE frequently convince themselves that it isn't much of a burden (as I had).

Having a chance to look at it with hindsight and not from the middle of it, I was very convinced that the POE way of providing cooperative multi-tasking is indeed quite a burden, even after one has gotten used to the awkwardness. Even the simplest algorithm, if it can involve a pause (such as reading from a network socket) must be broken into pieces. And any routine that calls such a fractured routine must itself be broken apart. And anything that calls that, etc.

This fracturing also means you don't have much of a call stack and can't easily throw and catch exceptions across the fractures. It keeps getting in your way every day in small and large ways.

So, I don't consider using POE to be an untenable solution. But I believe it incurs enough of a cost that I wouldn't make it my first choice.

I believe that forks.pm and Coro are worth considering first, as they allow unfractured coding. Coro will not use multiple "cores", but in most cases I don't mind because I'm usually either using a system that has plenty of other things to do or I have $N servers dedicated to the task and can just run 4 Coro-using instances on each quad-core server (for example). But the limitation is worth keeping in mind as there are certainly situations where it is a drawback.

- tye        

  • Comment on Re^7: Strange memory leak using just threads (forks.pm)

Replies are listed 'Best First'.
Re^8: Strange memory leak using just threads (forks.pm)
by BrowserUk (Patriarch) on Sep 21, 2010 at 05:30 UTC
    Hence the existence of forks.pm. It has the improved efficiency of creating a new "thread" because it uses fork() to do that
    1. How long does it take to start one thread per core? Answer: 0.0334138870239258 / 4.

      Sure, fork will be faster. But on any process long-running enough to be worth multi-threading, 0.01 per core is but a drop in the ocean.

    2. How long does it take to communicate 1000 work items to each of those work threads? Answer: 4 cores; 4 threads; 4000 items; 0.113806962966919 seconds.

      How long will it take to

      1. freeze those work items;
      2. send them via sockets;
      3. receive them at the other end;
      4. unfreeze them?

      And I bet that establishing those socket connections more than negates the lower start-up cost of fork.

    Which timing is the more important?

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

      How much will it cost to replace 4 with 64? With 128? With 256?

[reply]

        About 10 seconds to edit the test; much less to run it ( < 0.01s per core): (

        Updated: Improved the tests; de-obfuscated the code. 

        perl 
-Mthreads="stack_size,4096" 
-Mthreads::shared 
-MTime::HiRes=time 
-wE"
    $N = 64;
    $t = time;
    my$c :shared = 0;
    async( sub{ ++$c; sleep 10 } )->detach for 1..$N;
    1 while $c < $N;
    say time-$t;
    sleep 10
"
0.578999996185303

perl 
-Mthreads="stack_size,4096" 
-Mthreads::shared 
-MTime::HiRes=time 
-wE"
    $N = 256;
    $t = time;
    my $c :shared = 0;
    async( sub{ ++$c; sleep 10 } )->detach for 1..$N;
    1 while $c < $N;
    say time - $t;
    sleep 10
"
2.57699990272522

        [download]

        Editing the number is all that would be required.

        Now. How long will it take to re-tune your POE-behemoth based application when moving from running on 4 cores to 256 cores?

        From my experience of tuning event-driven systems for different hardware: weeks.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom