Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re^2: Code brewing for the upcoming MCE 10 year anniversary

by marioroy (Prior)
on Nov 04, 2022 at 10:55 UTC ( [id://11147961]=note: print w/replies, xml ) Need Help??


in reply to Re: Code brewing for the upcoming MCE 10 year anniversary
in thread Code brewing for the upcoming MCE 10 year anniversary

Now, I wonder about Python vs Perl. And then, something wonderful...

Python

from sys import stdout from multiprocessing import Process from multiprocessing import Barrier def task(barrier, number): for i in range(1, 400 + 1): print(f'{i}: {number}') barrier.wait() if __name__ == '__main__': stdout.flush() num_workers = 500 workers = list() # create a barrier barrier = Barrier(num_workers) # create the worker processes for i in range(1, num_workers + 1): workers.append(Process(target=task, args=(barrier, i))) workers[-1].start() # wait for all processes to finish for w in workers: w.join()

Perl

This seems odd, I know. Folks cannot run this. MCE::Simple will be released soon. I want to share the performance uptake having a limiter.

use MCE::Simple -strict, -signatures; use MCE::Barrier; sub task ($barrier, $number) { for (1 .. 400) { say "$_: $number"; $barrier->wait; } } sub main () { STDOUT->autoflush(1); my $num_workers = 500; # create a barrier my $barrier = MCE::Barrier->new($num_workers); # create the worker processes spawn task($barrier, $_) for 1 .. $num_workers; # wait for all processes to finish $_->join for MCE::Simple->list; # same as sync; } main();

Results

Python3 and Perl complete in 4.1 seconds. That was with the limiter commented out. The time is 1.5 seconds with the limiter enabled.

sub new { ... # [10] limiter (!$is_winenv && !$tid) ? MCE::Semaphore->new(3,1) : (), ... } sub _wait { ... $obj->[10]->down if $obj->[10]; # limiter my ($parties, $synced, $timed_out, $aborted) = $obj->_up($flag); $obj->[10]->up if $obj->[10]; ... }

The other oddity is that I cannot spin 1,000 processes in Python. It complains about running out of file handles. Perl, no problem!

Replies are listed 'Best First'.
Re^3: Code brewing for the upcoming MCE 10 year anniversary
by Fletch (Bishop) on Nov 04, 2022 at 16:12 UTC

    Using strace to watch what python is doing under the hood might give insight to the fd limit. Maybe it’s doing a dup or pipe behind the scenes for some sort of reason.

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

      On my box, the open files ulimit is 1024. No wonder !! Recently, I experienced the same issue running MCE with init_relay enabled. MCE::Relay requires MCE to configure a read-write channel per each worker.

Re^3: Code brewing for the upcoming MCE 10 year anniversary
by bliako (Monsignor) on Nov 04, 2022 at 15:52 UTC

    Congratulations for your MCE module.

     It complains about running out of file handles.

    I thought that was OS specific, unless P. sets its own limits too?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11147961]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (5)
As of 2024-04-19 06:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found