in reply to Re^7: Strange memory leak using just threads (forks.pm)
in thread Strange memory leak using just threads

Hence the existence of forks.pm. It has the improved efficiency of creating a new "thread" because it uses fork() to do that
  1. How long does it take to start one thread per core? Answer: 0.0334138870239258 / 4.

    Sure, fork will be faster. But on any process long-running enough to be worth multi-threading, 0.01 per core is but a drop in the ocean.

  2. How long does it take to communicate 1000 work items to each of those work threads? Answer: 4 cores; 4 threads; 4000 items; 0.113806962966919 seconds.

    How long will it take to

    1. freeze those work items;
    2. send them via sockets;
    3. receive them at the other end;
    4. unfreeze them?

    And I bet that establishing those socket connections more than negates the lower start-up cost of fork.

Which timing is the more important?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy
  • Comment on Re^8: Strange memory leak using just threads (forks.pm)

Replies are listed 'Best First'.
Re^9: Strange memory leak using just threads (forks.pm)
by zwon (Abbot) on Sep 21, 2010 at 05:54 UTC

    How much will it cost to replace 4 with 64? With 128? With 256?

[reply]

      About 10 seconds to edit the test; much less to run it ( < 0.01s per core): (

      Updated: Improved the tests; de-obfuscated the code. 

      perl 
-Mthreads="stack_size,4096" 
-Mthreads::shared 
-MTime::HiRes=time 
-wE"
    $N = 64;
    $t = time;
    my$c :shared = 0;
    async( sub{ ++$c; sleep 10 } )->detach for 1..$N;
    1 while $c < $N;
    say time-$t;
    sleep 10
"
0.578999996185303

perl 
-Mthreads="stack_size,4096" 
-Mthreads::shared 
-MTime::HiRes=time 
-wE"
    $N = 256;
    $t = time;
    my $c :shared = 0;
    async( sub{ ++$c; sleep 10 } )->detach for 1..$N;
    1 while $c < $N;
    say time - $t;
    sleep 10
"
2.57699990272522

      [download]

      Editing the number is all that would be required.

      Now. How long will it take to re-tune your POE-behemoth based application when moving from running on 4 cores to 256 cores?

      From my experience of tuning event-driven systems for different hardware: weeks.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

        I was talking about money actually - 256 cores system will be very expensive. But looking onto your example... from my experience it requires quite a lot of time to find the problem in the threaded application when somebody forget to lock shared variable before changing its value.

        Upd: And the tests are not correct BTW. Time required for creating new thread depends on the size of all existing variables.

[reply]

In Section Seekers of Perl Wisdom