Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^3: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run

by 1nickt (Canon)
on Apr 07, 2017 at 19:29 UTC ( [id://1187422]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
in thread Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run

Hello Mario,

(BTW you mentioned that having the signal handlers as well as END may be overdone ... the reason I have the data dumped in END is so it prints out upon uncaught exception.)

I made the change to MCE::Shared::Server as instructed. I took your last script (in the post I am replying to) and modified slightly. First, took some of the debug statements away and cleaned up others. Second, only dump the data in INT from parent process (as you have it in END). This gives following results:

With CTL-C, the END block is never reached by the parent; only the children. But the parent dumps the data in INT.

perl mce9.pl Parent PID 11581 worker 2 (11584) processing chunk 1 worker 1 (11583) processing chunk 2 worker 1 (11583) processing chunk 4 worker 2 (11584) processing chunk 3 ^CHello from INT: 11584 Hello from INT: 11581 Hello from END block: 11584 Parent in INT: $VAR1 = bless( { '00 11584' => '1491592596', '01 11583' => '1491592596', '02 11584' => '1491592598', '03 11583' => '1491592598' }, 'MCE::Shared::Hash' ); ## mce9.pl: caught signal (INT), exiting Hello from INT: 11583 Hello from END block: 11583 Killed
When running to completion, the parent reaches the END block and dumps the data there:
perl mce9.pl Parent PID 11554 worker 2 (11557) processing chunk 1 worker 1 (11556) processing chunk 2 worker 2 (11557) processing chunk 3 worker 1 (11556) processing chunk 4 worker 1 (11556) processing chunk 6 worker 2 (11557) processing chunk 5 worker 1 (11556) processing chunk 7 Hello from END block: 11557 Hello from END block: 11556 Hello from END block: 11554 Parent in END: $VAR1 = bless( { '00 11557' => '1491592580', '01 11556' => '1491592580', '02 11557' => '1491592582', '03 11556' => '1491592582', '04 11557' => '1491592584', '05 11556' => '1491592584', '06 11556' => '1491592586' }, 'MCE::Shared::Hash' );
Regarding END blocks, I believe Perl does guarantee the order (unlike BEGIN), which is Last In First Out. So the END block in the script should be executed before the one in a module that is loaded by use.

Code now:

use strict; use warnings; use feature 'say'; use Data::Dumper; ++$Data::Dumper::Sortkeys; use MCE::Loop; use MCE::Shared; $|++; my $pid = $$; say "Parent PID $pid"; my $hash = MCE::Shared->hash(); $SIG{'INT'} = $SIG{'TERM'} = sub { my $signal = shift; $SIG{'INT'} = $SIG{'TERM'} = sub {}; say "Hello from $signal: $$"; if ( $$ == $pid ) { say 'Parent in INT: ' . Dumper $hash->export; } MCE::Signal::stop_and_exit('INT'); }; MCE::Loop->init( max_workers => 2, chunk_size => 1, user_begin => sub { $SIG{'INT'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE->exit(0); }; $SIG{'TERM'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE::Signal::stop_and_exit($signal); }; } ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; say sprintf 'worker %s (%s) processing chunk %s', MCE->wid, MCE->p +id, $chunk_id; for ( @{ $chunk_ref } ) { $hash->{ sprintf '%.2d %s', $_, $$ } = time; sleep 2; } } ( 0 .. 6 ); MCE::Loop->finish; END { say "Hello from END block: $$"; if ( $$ == $pid ) { say 'Parent in END: ' . Dumper $hash->export; } }

Thank you again.


The way forward always starts with a minimal test.
  • Comment on Re^3: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
  • Select or Download Code

Replies are listed 'Best First'.
Re^4: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
by Anonymous Monk on Apr 07, 2017 at 20:38 UTC

    Thanks 1nickt. Regarding the use of the END block, no problem and why not use it. MCE::Signal is the reason for why the END block is not called for the parent process. It ends up doing a KILL signal. Fortunately, one can disable that by loading MCE::Signal before other MCE modules and pass the -no_kill9 option. Now that the END block is working, the handlers for the parent process are no longer needed. Nor the TERM handler for MCE workers. The script now looks like this.

    Regarding shared objects, having OO and auto-dereferencing on the fly makes it so natural versus calling tied(%hash)->method.

    use strict; use warnings; use feature 'say'; use Data::Dumper; ++$Data::Dumper::Sortkeys; use MCE::Signal qw( -no_kill9 ); use MCE::Loop; use MCE::Shared; $|++; my $pid = $$; say "Parent PID $pid"; my $hash = MCE::Shared->hash(); MCE::Loop->init( max_workers => 2, chunk_size => 1, user_begin => sub { $SIG{'INT'} = sub { my $signal = shift; say "Hello from $signal: $$"; MCE->exit(0); }; } ); mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; say sprintf 'worker %s (%s) processing chunk %s', MCE->wid, MCE->p +id, $chunk_id; for ( @{ $chunk_ref } ) { $hash->{ sprintf '%.2d %s', $_, $$ } = time; sleep 2; } } ( 0 .. 6 ); MCE::Loop->finish; END { say "Hello from END block: $$"; if ( $$ == $pid ) { say 'Parent in END: ' . Dumper $hash->export; } }

    I will test the change to MCE::Shared::Server. It's not feasible to handle both situations I'm not sure. However, it seems important for the shared-server to stick around longer to handle requests made inside handlers and END blocks.

    sub _loop { $_is_client = 0; # $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { # $SIG{INT} = $SIG{$_[0]} = sub { }; # # CORE::kill($_[0], $_is_MSWin32 ? -$$ : -getpgrp); # for my $_i (1..15) { sleep 0.060 } # # CORE::kill('KILL', $$); # CORE::exit(255); # }; $SIG{HUP} = $SIG{INT} = $SIG{QUIT} = $SIG{TERM} = sub { }; ... }

    It sure is nice to have the END block working. But it requires loading MCE::Signal qw( -no_kill9 ) before MCE modules to take effect. Signal handling is not fun. Likewise, parallel programming is crazy. But, with your help 1nickt, it's almost there.

    We're on the road. Amazingly Wifi from the laptop via the phone is working well.

    Thank you, 1nickt. Thank you, Perlmonks.

      Am validating the change for _loop in MCE::Shared::Server. Also, in _stop (around line 370), need to comment out two lines. Otherwise, the shared-manager process may linger around as a zombie process.

      sub _stop { return unless ($_is_client && $_init_pid && $_init_pid eq "$$.$_tid +"); # return if ($INC{'MCE/Signal.pm'} && $MCE::Signal::KILLED); # return if ($MCE::Shared::Server::KILLED); ... }

        Hi Mario,

        Yes, this code with the two changes to MCE::Shared::Server is working exactly as expected when run to completion, and when interrupted with CTL-C. Nice!

        For completion I wanted to test what would happen with an uncaught exception, so I added a fatal operation to the loop:

        for ( @{ $chunk_ref } ) { say $_ / 0 if $_ == 4;
        ... which resulted in the worker dying with the expected exception and message from Perl, while the manager and the other workers continued to completion, including updating the shared hash:
        perl mce10.pl Parent PID 29953 worker 2 (29957) processing chunk 1 worker 1 (29956) processing chunk 2 worker 2 (29957) processing chunk 3 worker 1 (29956) processing chunk 4 worker 1 (29956) processing chunk 6 worker 2 (29957) processing chunk 5 Illegal division by zero at mce10.pl line 27, <__ANONIO__> line 6. Hello from END block: 29957 worker 1 (29956) processing chunk 7 Hello from END block: 29956 Hello from END block: 29953 Parent in END: $VAR1 = bless( { '00 29957' => '1491661589', '01 29956' => '1491661589', '02 29957' => '1491661591', '03 29956' => '1491661591', '05 29956' => '1491661593', '06 29956' => '1491661595' }, 'MCE::Shared::Hash' );
        ... note the missing key for #4.

        I think that this is a good optional behaviour. But I think it would be nice to have the default case be: that an uncaught exception kills the whole program. One could choose to have the manager ignore an exception in a worker process, via a switch of some kind (maybe an option to MCE::Signal ?). But in that case I think it would be important to document the behaviour as demonstrated above, so users can know that the shared data cache will not necessarily contain all the expected data.

        So that by default one can count on: either the shared data structure being populated as expected, or an exception ... a partially-populated data structure should only be provided on demand and with a warning.

        Thank you again.


        The way forward always starts with a minimal test.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1187422]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (9)
As of 2024-04-18 08:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found