Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run

by Anonymous Monk
on Apr 07, 2017 at 18:13 UTC ( [id://1187417]=note: print w/replies, xml ) Need Help??


in reply to Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run

Okay, I see the issue. The shared-manager process is exiting from receiving the signal. Thus, not able to respond to later requests. Hence stalling the script.

Inside MCE::Shared::Server.pm around line 461, comment out the handler code and add the subsequent line. Basically, the shared-manager must still respond to requests inside application handlers. I'm not sure if I can handle both cases. This handler was placed here in the event something killed the shared-manager process.

sub _loop { $_is_client = 0; # $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { # $SIG{INT} = $SIG{$_[0]} = sub { }; # # CORE::kill($_[0], $_is_MSWin32 ? -$$ : -getpgrp); # for my $_i (1..15) { sleep 0.060 } # # CORE::kill('KILL', $$); # CORE::exit(255); # }; $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { }; ... }

I've simplyfied the handling for the MCE script.

use strict; use warnings; use feature 'say'; use Data::Dumper; ++$Data::Dumper::Sortkeys; use Time::HiRes 'sleep'; use MCE::Loop; use MCE::Shared; $|++; my $pid = $$; say "PID $pid"; my $hash = MCE::Shared->hash(); $SIG{'INT'} = $SIG{'TERM'} = sub { my $signal = shift; $SIG{'INT'} = $SIG{'TERM'} = sub {}; say "Hello from $signal: $$"; say 'Parent is ready to dump'; say 'Dumping: ' . Dumper $hash->export; MCE::Signal::stop_and_exit('INT'); }; MCE::Loop->init( max_workers => 2, chunk_size => 1, user_begin => sub { $SIG{'INT'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE->exit(0); }; $SIG{'TERM'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE::Signal::stop_and_exit($signal); }; } ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; say sprintf 'Forked worker in slot %s with pid %s for chunk %s', M +CE->wid, MCE->pid, $chunk_id; for ( @{ $chunk_ref } ) { $hash->{ sprintf '%.2d %s', $_, $$ } = time; say "After $_: " . Dumper $hash->export; sleep 2; } } ( 0 .. 12 ); MCE::Loop->finish; say "Parent is ready to dump"; say 'Dumping: ' . Dumper $hash->export;
  • Comment on Re: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
  • Select or Download Code

Replies are listed 'Best First'.
Re^2: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
by Anonymous Monk on Apr 07, 2017 at 18:27 UTC

    Ah, upon closer inspection, the reason the END block isn't called is from the KILL line.

    # $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { # $SIG{INT} = $SIG{$_[0]} = sub { }; # # CORE::kill($_[0], $_is_MSWin32 ? -$$ : -getpgrp); # for my $_i (1..15) { sleep 0.060 } # # CORE::kill('KILL', $$); # CORE::exit(255); # };

    There is no guarantee to which END block is called first by Perl. MCE::Shared has an END block to notify the shared-manager to exit. The script will stall had Perl called that one first. Therefore, leave intact the sig handling bits at the application level. The END block is not necessary. But simply added to see workers enter it.

    use strict; use warnings; use feature 'say'; use Data::Dumper; ++$Data::Dumper::Sortkeys; use Time::HiRes 'sleep'; use MCE::Loop; use MCE::Shared; $|++; my $pid = $$; say "PID $pid"; my $hash = MCE::Shared->hash(); $SIG{'INT'} = $SIG{'TERM'} = sub { my $signal = shift; $SIG{'INT'} = $SIG{'TERM'} = sub {}; say "Hello from $signal: $$"; say 'Parent is ready to dump'; say 'Dumping: ' . Dumper $hash->export; MCE::Signal::stop_and_exit('INT'); }; MCE::Loop->init( max_workers => 2, chunk_size => 1, user_begin => sub { $SIG{'INT'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE->exit(0); }; $SIG{'TERM'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE::Signal::stop_and_exit($signal); }; } ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; say sprintf 'Forked worker in slot %s with pid %s for chunk %s', M +CE->wid, MCE->pid, $chunk_id; for ( @{ $chunk_ref } ) { $hash->{ sprintf '%.2d %s', $_, $$ } = time; say "After $_: " . Dumper $hash->export; sleep 3; } } ( 0 .. 12 ); MCE::Loop->finish; END { say "Hello from END block: $$"; if ($$ == $pid) { say "Parent is ready to dump"; say 'Dumping: ' . Dumper $hash->export; } }

    I will make a new MCE::Shared update after more testing. Thank you, 1nickt.

      Hello Mario,

      (BTW you mentioned that having the signal handlers as well as END may be overdone ... the reason I have the data dumped in END is so it prints out upon uncaught exception.)

      I made the change to MCE::Shared::Server as instructed. I took your last script (in the post I am replying to) and modified slightly. First, took some of the debug statements away and cleaned up others. Second, only dump the data in INT from parent process (as you have it in END). This gives following results:

      With CTL-C, the END block is never reached by the parent; only the children. But the parent dumps the data in INT.

      perl mce9.pl Parent PID 11581 worker 2 (11584) processing chunk 1 worker 1 (11583) processing chunk 2 worker 1 (11583) processing chunk 4 worker 2 (11584) processing chunk 3 ^CHello from INT: 11584 Hello from INT: 11581 Hello from END block: 11584 Parent in INT: $VAR1 = bless( { '00 11584' => '1491592596', '01 11583' => '1491592596', '02 11584' => '1491592598', '03 11583' => '1491592598' }, 'MCE::Shared::Hash' ); ## mce9.pl: caught signal (INT), exiting Hello from INT: 11583 Hello from END block: 11583 Killed
      When running to completion, the parent reaches the END block and dumps the data there:
      perl mce9.pl Parent PID 11554 worker 2 (11557) processing chunk 1 worker 1 (11556) processing chunk 2 worker 2 (11557) processing chunk 3 worker 1 (11556) processing chunk 4 worker 1 (11556) processing chunk 6 worker 2 (11557) processing chunk 5 worker 1 (11556) processing chunk 7 Hello from END block: 11557 Hello from END block: 11556 Hello from END block: 11554 Parent in END: $VAR1 = bless( { '00 11557' => '1491592580', '01 11556' => '1491592580', '02 11557' => '1491592582', '03 11556' => '1491592582', '04 11557' => '1491592584', '05 11556' => '1491592584', '06 11556' => '1491592586' }, 'MCE::Shared::Hash' );
      Regarding END blocks, I believe Perl does guarantee the order (unlike BEGIN), which is Last In First Out. So the END block in the script should be executed before the one in a module that is loaded by use.

      Code now:

      use strict; use warnings; use feature 'say'; use Data::Dumper; ++$Data::Dumper::Sortkeys; use MCE::Loop; use MCE::Shared; $|++; my $pid = $$; say "Parent PID $pid"; my $hash = MCE::Shared->hash(); $SIG{'INT'} = $SIG{'TERM'} = sub { my $signal = shift; $SIG{'INT'} = $SIG{'TERM'} = sub {}; say "Hello from $signal: $$"; if ( $$ == $pid ) { say 'Parent in INT: ' . Dumper $hash->export; } MCE::Signal::stop_and_exit('INT'); }; MCE::Loop->init( max_workers => 2, chunk_size => 1, user_begin => sub { $SIG{'INT'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE->exit(0); }; $SIG{'TERM'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE::Signal::stop_and_exit($signal); }; } ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; say sprintf 'worker %s (%s) processing chunk %s', MCE->wid, MCE->p +id, $chunk_id; for ( @{ $chunk_ref } ) { $hash->{ sprintf '%.2d %s', $_, $$ } = time; sleep 2; } } ( 0 .. 6 ); MCE::Loop->finish; END { say "Hello from END block: $$"; if ( $$ == $pid ) { say 'Parent in END: ' . Dumper $hash->export; } }

      Thank you again.


      The way forward always starts with a minimal test.

        Thanks 1nickt. Regarding the use of the END block, no problem and why not use it. MCE::Signal is the reason for why the END block is not called for the parent process. It ends up doing a KILL signal. Fortunately, one can disable that by loading MCE::Signal before other MCE modules and pass the -no_kill9 option. Now that the END block is working, the handlers for the parent process are no longer needed. Nor the TERM handler for MCE workers. The script now looks like this.

        Regarding shared objects, having OO and auto-dereferencing on the fly makes it so natural versus calling tied(%hash)->method.

        use strict; use warnings; use feature 'say'; use Data::Dumper; ++$Data::Dumper::Sortkeys; use MCE::Signal qw( -no_kill9 ); use MCE::Loop; use MCE::Shared; $|++; my $pid = $$; say "Parent PID $pid"; my $hash = MCE::Shared->hash(); MCE::Loop->init( max_workers => 2, chunk_size => 1, user_begin => sub { $SIG{'INT'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE->exit(0); }; } ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; say sprintf 'worker %s (%s) processing chunk %s', MCE->wid, MCE->p +id, $chunk_id; for ( @{ $chunk_ref } ) { $hash->{ sprintf '%.2d %s', $_, $$ } = time; sleep 2; } } ( 0 .. 6 ); MCE::Loop->finish; END { say "Hello from END block: $$"; if ( $$ == $pid ) { say 'Parent in END: ' . Dumper $hash->export; } }

        I will test the change to MCE::Shared::Server. It's not feasible to handle both situations I'm not sure. However, it seems important for the shared-server to stick around longer to handle requests made inside handlers and END blocks.

        sub _loop { $_is_client = 0; # $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { # $SIG{INT} = $SIG{$_[0]} = sub { }; # # CORE::kill($_[0], $_is_MSWin32 ? -$$ : -getpgrp); # for my $_i (1..15) { sleep 0.060 } # # CORE::kill('KILL', $$); # CORE::exit(255); # }; $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { }; ... }

        It sure is nice to have the END block working. But it requires loading MCE::Signal qw( -no_kill9 ) before MCE modules to take effect. Signal handling is not fun. Likewise, parallel programming is crazy. But, with your help 1nickt, it's almost there.

        We're on the road. Amazingly Wifi from the laptop via the phone is working well.

        Thank you, 1nickt. Thank you, Perlmonks.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1187417]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (7)
As of 2024-04-23 10:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found