Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^3: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run

by Anonymous Monk
on Apr 07, 2017 at 15:41 UTC ( [id://1187409]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
in thread Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run

Why, when chunk_size is set to `1`, is the same process reused?

Workers persist from start to finish. chunk_size refers to how many items a given worker receives per user_func.

user_begin
user_func
user_func
user_func
...
user_func
user_end

[download]

Why, when the interrupt is given, is nothing inside the SIG handlers being executed?

The overriding of SIG handlers must be done before calling mce_loop or before workers are spawned.

What is the purpose of emptying the hash in the INT signal handler?

The worker or parent receiving the signal displays the content and subsequently clears the hash before notifying others to exit. Thus, causing other workers to call the handler. We only need to display the content once.

  • Comment on Re^3: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
  • Download Code

Replies are listed 'Best First'.
Re^4: Interrupt multi-process program while using MCE::Shared hash: END block code does not (all) run
by Anonymous Monk on Apr 07, 2017 at 15:53 UTC

    Also, dumper traverses the hash. A mutex is likely necessary around Dumper.

    use strict; use warnings; use feature 'say';
use Data::Dumper; ++$Data::Dumper::Sortkeys;
use MCE::Shared;
use MCE::Loop;
$|++;

my $pid = $$; say "PID $pid";
my $mutex = MCE::Mutex->new();

tie my %hash, 'MCE::Shared', ();

$SIG{'INT'} = sub {
    my $signal = shift;
    say 'Hello from INT';
    if ( tied(%hash)->len ) {
        (MCE->wid == 0) ? say 'Parent is ready to dump'
                        : say 'Worker is ready to dump';

        say 'Dumping: ' . Dumper \%hash;
        %hash = ();
    }
    MCE::Signal::stop_and_exit($signal);
};

$SIG{'TERM'} = sub {
    my $signal = shift;
    say 'Hello from TERM';
    if (MCE->wid > 0) { # worker
        MCE->exit(0);
    } else { # parent
        say 'Parent is ready to dump';
        say 'Dumping: ' . Dumper \%hash;
        MCE::Signal::stop_and_exit($signal);
    }
};

MCE::Loop->init( max_workers => 2, chunk_size => 1 );

mce_loop {
    my ( $mce, $chunk_ref, $chunk_id ) = @_;
    say sprintf 'Forked worker in slot %s with pid %s for chunk %s', M
+CE->wid, MCE->pid, $chunk_id;
    for ( @{ $chunk_ref } ) {
        $hash{ sprintf '%.2d %s', $_, $$ } = time;
        $mutex->synchronize( sub {
           say "After $_: " . Dumper \%hash;
        });
        sleep 1;
    }
} ( 0 .. 4 );

MCE::Loop->finish;

END {
    say sprintf '%s %s (%s) in END', $$, time, $$ == $pid ? 'Parent' :
+ 'Child';
    if ( MCE->wid == 0 or $$ == $pid ) {
        say "Parent is ready to dump";
        say 'Dumping: ' . Dumper \%hash;
    }
}

    [download]
[reply]
[d/l]

      Update: Removing the shared hash code solves the hanging process problem, see below.

      Unfortunately this, too, simply hangs on interrupt: 

      PID 28907
Forked worker in slot 2 with pid 28910 for chunk 1
Forked worker in slot 1 with pid 28909 for chunk 2
After 1: $VAR1 = {
          '00 28910' => '1491580974',
          '01 28909' => '1491580974'
        };

After 0: $VAR1 = {
          '00 28910' => '1491580974',
          '01 28909' => '1491580974'
        };

Forked worker in slot 1 with pid 28909 for chunk 3
Forked worker in slot 2 with pid 28910 for chunk 4
After 2: $VAR1 = {
          '00 28910' => '1491580974',
          '01 28909' => '1491580974',
          '02 28909' => '1491580975'
        };

After 3: $VAR1 = {
          '00 28910' => '1491580974',
          '01 28909' => '1491580974',
          '02 28909' => '1491580975',
          '03 28910' => '1491580975'
        };

^CHello from INT
Hello from INT
Hello from INT

      [download]
      ... hangs here ... all three processes alive: 
      pgr mce

nickt    28907       01:26 perl mce4.pl
nickt    28909       01:26 perl mce4.pl
nickt    28910       01:26 perl mce4.pl

      [download]

      I am running on Debian, Perl 5.16.3, with the very latest MCE modules.

      Thank you.

      Update: Note that if I simply comment out all the shared hash code, the program can be interrupted and the processes do not hang, but the END block is never reached: 

      perl mce4.pl

PID 29593
Forked worker in slot 2 with pid 29596 for chunk 1
Forked worker in slot 1 with pid 29595 for chunk 2
Forked worker in slot 1 with pid 29595 for chunk 4
Forked worker in slot 2 with pid 29596 for chunk 3
^CHello from INT
Worker is ready to dump
Hello from INT
Parent is ready to dump
Hello from INT
Worker is ready to dump

## mce4.pl: caught signal (INT), exiting

Killed

      [download]
      ... above with this code: 
      use strict; use warnings; use feature 'say';
use Data::Dumper; ++$Data::Dumper::Sortkeys;
use MCE::Shared;
use MCE::Loop;
$|++;

my $pid = $$; say "PID $pid";
my $mutex = MCE::Mutex->new();

#tie my %hash, 'MCE::Shared', ();

$SIG{'INT'} = sub {
    my $signal = shift;
    say 'Hello from INT';
#    if ( tied(%hash)->len ) {
        (MCE->wid == 0) ? say 'Parent is ready to dump'
                        : say 'Worker is ready to dump';

#        say 'Dumping: ' . Dumper \%hash;
#        %hash = ();
#    }
    MCE::Signal::stop_and_exit($signal);
};

$SIG{'TERM'} = sub {
    my $signal = shift;
    say 'Hello from TERM';
    if (MCE->wid > 0) { # worker
        MCE->exit(0);
    } else { # parent
        say 'Parent is ready to dump';
#        say 'Dumping: ' . Dumper \%hash;
        MCE::Signal::stop_and_exit($signal);
    }
};

MCE::Loop->init( max_workers => 2, chunk_size => 1 );

mce_loop {
    my ( $mce, $chunk_ref, $chunk_id ) = @_;
    say sprintf 'Forked worker in slot %s with pid %s for chunk %s', M
+CE->wid, MCE->pid, $chunk_id;
    for ( @{ $chunk_ref } ) {
#        $hash{ sprintf '%.2d %s', $_, $$ } = time;
        $mutex->synchronize( sub {
#           say "After $_: " . Dumper \%hash;
        });
        sleep 1;
    }
} ( 0 .. 4 );

MCE::Loop->finish;

END {
    say sprintf '%s %s (%s) in END', $$, time, $$ == $pid ? 'Parent' :
+ 'Child';
    if ( MCE->wid == 0 or $$ == $pid ) {
        say "Parent is ready to dump";
#        say 'Dumping: ' . Dumper \%hash;
    }
}

      [download]


      The way forward always starts with a minimal test.
[reply]
[d/l]
[select]

        Yes, hangs when pressing Control-C.

        Off topic, the OO interface is so nice. The mutex is not necessary. Btw, it is a bit much to have both the handlers and the END block. Only one is needed. But it still hangs when pressing Ctrl-C. However, sending INT from another terminal works fine. For this post, wanted to remove the TIE interface. Instead, access the shared hash via the OO interface. MCE::Shared can handle dereferencing automatically for setting the value. Thus, giving you both OO and dereferencing when needed.

        use strict; use warnings; use feature 'say';
use Data::Dumper; ++$Data::Dumper::Sortkeys;
use MCE::Shared;
use MCE::Loop;
$|++;

my $pid = $$; say "PID $pid";
my $hash = MCE::Shared->hash();

$SIG{'INT'} = sub {
    my $signal = shift; $SIG{'INT'} = $SIG{'TERM'} = sub { };
    say "Hello from $signal: $$";
    if ( $hash->len ) {
        ( MCE->wid == 0 ) ? say 'Parent is ready to dump'
                          : say 'Worker is ready to dump';

        say 'Dumping: ' . Dumper $hash->export;
        $hash->clear;
    }

    MCE::Signal::stop_and_exit($signal);
};

$SIG{'TERM'} = sub {
    my $signal = shift; $SIG{'INT'} = $SIG{'TERM'} = sub { };
    say "Hello from $signal: $$";
    if (MCE->wid) { # worker
        MCE->exit(0);
    } else { # parent
        say 'Parent is ready to dump';
        say 'Dumping: ' . Dumper $hash->export;
        MCE::Signal::stop_and_exit($signal);
    }
};

MCE::Loop->init( max_workers => 2, chunk_size => 1 );

mce_loop {
    my ( $mce, $chunk_ref, $chunk_id ) = @_;
    say sprintf 'Forked worker in slot %s with pid %s for chunk %s', M
+CE->wid, MCE->pid, $chunk_id;
    for ( @{ $chunk_ref } ) {
        $hash->{ sprintf '%.2d %s', $_, $$ } = time;
        say "After $_: " . Dumper $hash->export;
        sleep 3;
    }
} ( 0 .. 12 );

MCE::Loop->finish;

say "Parent is ready to dump";
say 'Dumping: ' . Dumper $hash->export;

        [download]

        There is the main process, workers, and also the shared-manager process. Something is stalling somewhere.

[reply]
[d/l]

        The stall is happening from inside the handlers. Commenting out the hash->len and associated block inside the handlers resolves the stall. But then, we're back to square one in wanting to display the contents.

        I'm on the road, my dad has fallen ill. I'm not myself lately. Am accepting that life is shorter each day. We only have little time with our parents. Your example is great. Signal handling is not fun. I do not understand why the END block isn't called. Will try setting the handlers to DEFAULT and other things including having the END block.

        Thank you 1nickt for this test case.

[reply]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://1187409]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others admiring the Monastery: (7)
As of 2024-04-24 09:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found