in reply to Perl threads loss of performance with system call

How does this perform on your box?

#!/usr/bin/env perl use strict; use warnings; use threads; use MCE::Hobo; use feature qw(say); use constant AMOUNT => 0.001; my $cores = MCE::Util::get_ncpu(); MCE::Hobo->init( max_workers => $cores ); for my $id ( 1 .. 50 ) { my $hobo = MCE::Hobo->create( \&mysub, $id ); say qq($id ) . $hobo -> pid(); } # MCE::Hobo->wait_all(); sub mysub { # my $id = shift; # say $id; for ( 1 .. 100 ) { qx( sleep @{[ AMOUNT ]};) } } __END__

«The Crux of the Biscuit is the Apostrophe»

Replies are listed 'Best First'.
Re^2: Perl threads loss of performance with system call
by Anonymous Monk on Jul 14, 2021 at 05:54 UTC

    Please find 2 examples, based on karlgoethebier's demonstration.

    Example 1:

    The posix_exit Hobo option is beneficial on Unix platforms. Be sure to close any file handles opened by the worker. The void_context option is helpful to reduce IPC overhead if not wanting result from workers.

    #!/usr/bin/env perl use strict; use warnings; use MCE::Hobo; use feature qw(say); use constant AMOUNT => 0.001; my $cores = MCE::Util::get_ncpu(); MCE::Hobo->init( max_workers => $cores, void_context => 1, posix_exit => 1, ); sub consumer_task { my $id = shift; say qq($id ) . MCE::Hobo->pid(); qx( sleep @{[ AMOUNT ]} ); } for my $id ( 1..800 ) { my $hobo = MCE::Hobo->create( \&consumer_task, $id ); } MCE::Hobo->wait_all();

    Example 2:

    Here, we minimize the spawning of Hobo workers and instead use a channel or a queue to send/receive input.

    #!/usr/bin/env perl use strict; use warnings; use MCE::Hobo; use MCE::Channel; use feature qw(say); use constant AMOUNT => 0.001; my $cores = MCE::Util::get_ncpu(); my $queue = MCE::Channel->new(); MCE::Hobo->init( void_context => 1, posix_exit => 1, ); # Consumers sub consumer_task { while ( my $input = $queue->dequeue() ) { say qq($input->{id} ) . MCE::Hobo->pid(); qx( sleep @{[ AMOUNT ]};) } } MCE::Hobo->create( \&consumer_task ) for 1..$cores; # Producer for ( 1..800 ) { $queue->enqueue({ id => $_ }); } $queue->end(); MCE::Hobo->wait_all();

      The channel object is bidirectional. Therefore, the worker can send its PID value to the producer. This has the effect of max one item in the channel. The producer writes to STDOUT like the demonstration by karlgoethebier.

      #!/usr/bin/env perl use strict; use warnings; use MCE::Hobo; use MCE::Channel; use feature qw(say); use constant AMOUNT => 0.001; my $cores = MCE::Util::get_ncpu(); my $queue = MCE::Channel->new(); MCE::Hobo->init( void_context => 1, posix_exit => 1, ); # Consumers sub consumer_task { while ( my $input = $queue->recv() ) { my $id = $input->{id}; $queue->send2( MCE::Hobo->pid() ); qx( sleep @{[ AMOUNT ]};) } } MCE::Hobo->create( \&consumer_task ) for 1..$cores; # Producer for my $id ( 1..800 ) { $queue->send({ id => $id }); my $hobo_pid = $queue->recv2(); say qq($id ) . $hobo_pid; } $queue->end(); MCE::Hobo->wait_all();

      Thanks. This looks good. What I never understood is why to call  MCE::Hobo->wait_all(); in this context. As it seems that all the hobos terminate correctly in this fubar loop without the call.

      «The Crux of the Biscuit is the Apostrophe»

Re^2: Perl threads loss of performance with system call
by daniel85 (Novice) on Jul 11, 2021 at 17:29 UTC
    Quite smoothly, but it exits before everything is terminated

      Strange. On my Mac it works as designed. 50 threads and as I have 12 cores 4 x 12 plus 2. It can be seen in the output as well as in the process table. For the moment I’m at a loss.

      «The Crux of the Biscuit is the Apostrophe»

      Uncommenting the wait_all line is necessary to ensure workers have completed and exited.

      MCE::Hobo->wait_all();

        No. It isn’t. Try it. I was quite surprised about the result.

        «The Crux of the Biscuit is the Apostrophe»