http://qs1969.pair.com?node_id=997094

DanEllison has asked for the wisdom of the Perl Monks concerning the following question:

I have a multithreaded application which spawns threads to track work. The single parent thread creates all the children threads. Each child thread executes an external command and then captures the output via an open on a pipe. If the child thread doesn't complete within some time limit, the parent sends the child a signal, and the child in turn attempts to kill the external process.

I'm not sure if its the fact that its windows and emulated, or if its due to some rule regarding waiting on I/O, but the child thread doesn't appear to be acting immediately on the signals. When I interupt the parent, then the child thread suddenly reports the earlier signals.

I don't want to SIGKILL the child thread as I want it to report what it had accomplished to that point. However, I set it up so that if I sent it a NUM16, it would do a SIGKILL on the external command. Since this didn't have any effect, it appears that its the child thread that is not responding, not the external command.

Maybe there's a totally better way of doing this.... Following is a snippet of the child thread code.

$self->msg('execute', dir => Cwd::getcwd(), cmd => $self->{cmd}, 'eval +' => $cmd); my $pid = open EXEC, "-|", "($cmd)2>&1" or die "Could not start comman +d."; foreach ( qw{ ABRT ALRM BREAK FPE HUP ILL INT PIPE QUIT SEGV STOP TERM + } ) { $SIG{$_} = sub { warn "$self->{-full}: Signal $_ caught"; $self->msg +('signal', signal => $_[0]); kill $_[0], $pid; }; } foreach ( qw{ USR1 NUM16 } ) { $SIG{$_} = sub { warn "$self->{-full}: Signal $_ caught"; $self->msg +('signal', signal => 'KILL'); kill 'KILL', $pid; }; } while (<EXEC>) { chomp; $self->msg('execute', output => $_); } close EXEC; $self->set(status => $?, stop => Scheduler::Time::utc); foreach ( qw{ ABRT ALRM BREAK FPE HUP ILL INT PIPE QUIT SEGV STOP TERM + USR1 NUM16 } ) { $SIG{$_} = 'DEFAULT'; };

Replies are listed 'Best First'.
Re: Signals vs. Windows
by BrowserUk (Patriarch) on Oct 03, 2012 at 21:18 UTC
    1. Signals and Windows:

      Not OS supported. Very limited emulation by Perl. Not reliable.

    2. Signals and threads:

      Signals are, at best, on platforms that support them natively, a very crude form of IPC.

      They are not a useful mechanism of Inter-Thread Communications.

      From the Linux man pages:

      The signal disposition is a per-process attribute: in a multithreaded application, the disposition of a particular signal is the same for all threads.

      A child created via fork(2) inherits a copy of its parent's signal dispositions. During an execve(2), the dispositions of handled signals are reset to the default; the dispositions of ignored signals are left unchanged.

      A signal may be generated (and thus pending) for a process as a whole (e.g., when sent using kill(2)) or for a specific thread (e.g., certain signals, such as SIGSEGV and SIGFPE, generated as a consequence of executing a specific machine-language instruction are thread directed, as are signals targeted at a specific thread using pthread_kill(3)). A process-directed signal may be delivered to any one of the threads that does not currently have the signal blocked. If more than one of the threads has the signal unblocked, then the kernel chooses an arbitrary thread to which to deliver the signal.

    3. Signals and blocking-IO:

      Since 5.8, Perl only delivers signals after the currently executing opcode completes (SAFE_SIGNALS). If the currently executing opcode is a blocking IO operation that may never complete -- eg. reading a line from a pipe -- the signal may never be delivered

    4. Sending every known signal:

      Utter desperation. :)

    For a solution to your problem that works, see Re^3: trying to get timeout to work (easier with threads).


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

      Signals are, at best, on platforms that support them natively, a very crude form of IPC.

      Yes, everyone is quick to suggest that, and yet not offer any working alternatives. Shared memory is not always an option. Port and pipe communication is excessive.

      If my entire process is being terminated from the keyboard, by a scheduler, or the O/S, a signal is exactly what I'm going to get, and I want to cleanup properly which it does. I was just trying to use the same logic internally to force the cleanup based on a timer.

      Sending every known signal: Utter desperation. :)

      Sending no, capturing yes. I wasn't sending every signal but one, maybe two. Understanding how the app responds to many different situations is just good testing.

      For a solution to your problem that works, see Re^3: trying to get timeout to work (easier with threads).

      Note that your example still uses the devilish signal. But yes, this seems to be the most promising suggestion yet. Resolves the blocking I/O by pushing it another thread deeper. Thank you.

        Note that your example still uses the devilish signal.

        Yes, but in the way it was intended -- as an IPC mechanism, not ITC.

        Resolves the blocking I/O by pushing it another thread deeper. Thank you.

        Indeed. (Be sure to look at the stack_size parameter to threads.)

        But, it isn't actually necessary to go one thread deeper. If your current main/parent thread can attempt to send a signal to its child threads to terminate them, it could also send that signal directly to the processes those child threads are waiting on, thus cutting out the middle man and the need for an extra thread.

        All that would be required is to shared the pids of those processes with the parent thread so that it knows where to send the signal when the time is right.

        The following script starts 5 asynchronous, piped subprocesses in threads that will each run for 5 seconds producing one 'line' of output per second. The threads share the pids of those processes with the main thread via a shared array indexed by their thread ids.

        The main thread then applies timeouts to those processes ranging from 3 to 7 seconds, killing them after the appropriate delay. It then gathers whatever output they had produced before the timeout from the threads by joining them.

        You can see -- in the console output after the __END__ -- that the first two processes were terminated (SIGINT) after 3 and 4 seconds respectively and their output was truncated. The other 3 complete their output and terminate normally.

        #! perl -slw use strict; use Time::HiRes qw[ sleep time ]; use threads; use threads::shared; my @pids :shared; sub asyncPipedProc { my $tid = threads->tid; my $cmd = shift; $pids[ $tid ] = open my $pipe, '-|', $cmd or die $!; my @input; push @input, $_ while <$pipe>; return @input; } my $cmd = q[ perl -E"$|++; sleep(1), say for 1 .. 5" ]; my @running; for my $timeout ( 3 .. 7 ) { push @running, [ threads->new( \&asyncPipedProc, $cmd, $_ ), time() + $timeout ]; } while( @running and sleep 0.1 ) { for my $i ( reverse 0 .. $#running ) { if( $running[ $i ][ 1 ] < time() ) { kill 2, $pids[ $running[ $i ][ 0 ]->tid ]; printf "From thread %d got:\n%s", $running[ $i ][ 0 ]->tid, join'', $running[ $i ][ 0 ]->join; splice @running, $i, 1; } } } __END__ C:\test>multi-timeout.pl Terminating on signal SIGINT(2) From thread 1 got: 1 2 3 Terminating on signal SIGINT(2) From thread 2 got: 1 2 3 4 From thread 3 got: 1 2 3 4 5 From thread 4 got: 1 2 3 4 5 From thread 5 got: 1 2 3 4 5

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

Re: Signals vs. Windows
by eyepopslikeamosquito (Archbishop) on Oct 03, 2012 at 20:57 UTC

    If the child thread doesn't complete within some time limit, the parent sends the child a signal, and the child in turn attempts to kill the external process.
    If this is Windows-only code, and the intent is to run an external command with a time limit, I suggest you take a look at the Win32::Job module. For a simple example of using it, see Re: Timing Windows commands.

    Since signals don't play nicely with threads generally and since signals are not part of the native Win32 API, I would stay away from signals for your application.

      No, this is not Windows only.
Re: Signals vs. Windows
by Anonymous Monk on Oct 03, 2012 at 16:38 UTC

      The pertinent text I missed was actually out of "threads" which was:

      Correspondingly, sending a signal to a thread does not disrupt the operation the thread is currently working on: The signal will be acted upon after the current operation has completed. For instance, if the thread is stuck on an I/O call, sending it a signal will not cause the I/O call to be interrupted such that the signal is acted up immediately.

      Nothing in the documentation suggests that signals are not the way to perform IPC, but simply threads don't react to signals as well as processes. Keep in mind, I don't want to simply suspend or terminate the thread, but interrupt it and send it off in another direction. So, the problem restated is, I guess, how do I interrupt a thread from an I/O operation?

        So, the problem restated is, I guess, how do I interrupt a thread from an I/O operation?

        1. dont use blocking I/O, use overlapped/async I/O

        2. use Native API, create an alertable *synchronous* file handle using NtCreateFile, post a timer APC to the blocked thread with SetWaitableTimer, the APC will call CancelIo from within the blocked thread, and the blocked I/O will unblock.

        Nothing in the documentation suggests that signals are not the way to perform IPC

        Hmm, "unsafe" about says it all :)

        So, the problem restated is, I guess, how do I interrupt a thread from an I/O operation?

        stop using safe signals, use unsafe signals, use kill

Re: Signals vs. Windows
by nikosv (Deacon) on Oct 04, 2012 at 10:34 UTC

    Windows work with messages while signals are emulated in console mode.

    Aside from that,on Windows for thread synchronization or notification you can you use events and WaitForMultipleObjects or WaitForMultipleObjectsEx. You can use the Win32::Event to get to them from Perl

    Each child thread can work and poll on when the event is signalled and abort if it does.

    Another advantage is that they are kernel events so they can pertain across processes too.

      All very well and good; but none of it helps the OP timeout a blocking read.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      RIP Neil Armstrong

        I see,so you mean that the child thread can't poll. What if he combines polling the event with non blocking IO ?