87C751 has asked for the wisdom of the Perl Monks concerning the following question:

HNY, oh brothers of accumulated wisdom!

I need to control two external programs to accomplish a stream transcoding application. App 1 obtains its input from a URL and writes to STDOUT. It also forks 2-3 children. App 2 reads from STDIN and I read its output and send it on its way.

The good old open(FH,"app1 URL | app2 |"); performs the transcoding, but the catch is that I need to kill the children when the consumer quits drinking from my output. (recall that input is a stream... it doesn't end by itself) close(FH) doesn't seem to kill the kids (or even return, for that matter).

$pid = open2($rdhndl,$wrhndl,"app1 URL | app2"); also transcodes, but the PID returned is that of the sh process running the pipeline and killing that PID doesn't kill the children of App 1 (and indeed doesn't always kill App 1).

system("killall app1"); would work (sort of), but it's ugly, inefficient and doesn't allow for multiple clients of my transcoder.

Combining separate open2() calls hasn't worked. I can't seem to structure the calls so that App 2 reads from App 1's output without intermediation. (OK, I couldn't get it to work at all)

App 2 will die politely when its input disappears, so I really only need to reliably kill the first App 1 process.

What's a coder to do?

Replies are listed 'Best First'.
Re: Taming multiple external processes
by Zaxo (Archbishop) on Jan 02, 2004 at 06:05 UTC

    Killing the shell process should send SIGHUP to its children. The default action for HUP is to terminate, but if it is not doing that you can set %SIG when you fork:

    my $pid; { local $SIG{HUP} = $SIG{INT}; $pid = open2($rdhndl,$wrhndl,"app1 URL | app2") or die $!; }
    What syntax are you using for kill? You certainly don't need to shell out to raise a signal in Perl.

    After Compline,
    Zaxo

      I'm using kill 1, $pid;, and I've tried all of 1, 9, HUP, INT, SIGHUP, SIGINT and SIGKILL, all with the same result. Even setting local $SIG{HUP} = $SIG{INT}; made no change. Each time, app2 dies and the sh process becomes a zombie, but I still have 4 app1 processes. If I then kill the server that spawned them, all of the processes vanish.

      Here's a ps list after a client has connected:

      7314 pts/1 S 0:00 /usr/bin/perl -w ./testserver 7315 pts/1 S 0:00 sh -c ogg123 -q -d raw -f - http://cal.icec +ast.net:8630/prog1.ogg | lame --silent -b 320 -r - 7316 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg 7317 pts/1 S 0:00 lame --silent -b 320 -r -x - - 7318 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg 7319 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg 7321 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg

      After the client disconnects, the server does kill 1, $pid on the pid returned from open2. The ps list now looks like this:

      7314 pts/1 S 0:00 /usr/bin/perl -w ./testserver 7315 pts/1 Z 0:00 [sh] <defunct> 7316 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg 7318 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg 7319 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg 7321 pts/1 S 0:00 ogg123 -q -d raw -f - http://cal.icecast.ne +t:8630/prog1.ogg

      You see that lame (my app2) is gone, and the shell is zombied. But all the ogg123 (app1) processes are still alive (and, according to the activity lights on the switch, still sucking data from the URL).

      Am I going mad?

Re: Taming multiple external processes
by tilly (Archbishop) on Jan 02, 2004 at 08:06 UTC
    Here is a demonstration of how to set up a pipeline like the one that you want which will give you the pid of the first process in the pipeline. Note that it is slightly complex and is easy to get wrong (I'm going out of my way to make sure that Perl spawns no shells to interpret shell commands):
    #! /usr/bin/perl use strict; use IPC::Open2; local *PIPE; local *OUT; my $pid = open(PIPE, "-|", "echo", "hi") or die "Can't echo: $!"; print "The pid of the echo command is: $pid"; <STDIN>; open2(\*OUT, "<&PIPE", "perl", "-ne", "print uc") or die "open2 failed: $!"; print while <OUT>;
    You can use ps to verify that before you tie the two programs together the pid that you got from open actually is the pid of the echo process that you launched.

    UPDATE: I left out the assignment to pid. Added. (I had included an earlier version of the code, then commented about a later one that demonstrated the pid, then partially updated the code but missed one line. My apologies for any confusion.)

      Ah... the $pid assignment did get frobbed. After re-adding that, your example works fine. But applying it to my task fails. The server process appears to hang on the open2 call. The child process is started, according to ps, but never starts reading input (according to its STDERR output). When I kill the open() process, the whole server dies with open2: close(main::PIPE) failed:  at ./testserver3 line 30, which seems to say that the PIPE handle couldn't be closed in the parent.

      This is somewhat frustrating.

        You may have copied part of it slightly wrong. When I used a local variable and a named string in open2, it really was important that I did so.

        If you read the documentation for IPC::Open2 you'll see that I'm trying to get Perl to dup the filehandle directly. That is important because a close will fail there because your first process already has unread data that it wants PIPE to read before it exits.

        If that isn't the problem, then you should make your own private copy of IPC::Open3, be sure that you are loading that version (eg with a use lib), and then start inserting debugging statements to figure out where and when it is doing the close.

        Also I will warn you. Depending on how app1 is coded, figuring out how to kill it might not work for you. After all when it forks, its kids might be talking to their STDOUT, which means that the second process in the pipeline is still being talked to. And if the kids ignore the signal when dad dies...

        Another thing to re-check. Does app1 really talk on STDOUT? If it talks on STDERR, or changes its behaviour when it finds itself not on a terminal, it could be very confusing. The output of app1 arguments here | cat 2> /dev/null at the shell might be revealing. (That calls it with output hooked to a pipe, and hides STDERR.)

      Did this example get frobbed in the posting? $pid is never assigned to or declared, and that <STDIN>; all by itself looks mighty strange. Oh, and it doesn't seem to work. :)

      I think I see what you're aiming at, though.

Re: Taming multiple external processes
by sgifford (Prior) on Jan 02, 2004 at 09:56 UTC

    Here's some code that will let you get the PID of each step in the pipeline. To avoid running a shell, pass each argument as a seperate parameter to pipeline.

    pipeline takes as its first parameter the filehandle where the command's standard input should come from, and the rest of the parameters are the command to run. It returns a 2-element list, with the PID and output filehandle of the command. With your example, you'd do something like:

    my($app1_pid,$app1_out)=pipeline(STDIN,'app1',$URL); my($app2_pid,$app2_out)=pipeline($app1_out,'app2'); while (<$app2_out>) { ... }

    Here's the pipeline code, and a short example that runs sort |uniq -c |sort -rn. It's only tested slightly, so it may need some tweaking.

    #!/usr/bin/perl use POSIX; my($pid1,$outfh1)=pipeline(STDIN,"sort"); my($pid2,$outfh2)=pipeline($outfh1,"uniq -c"); my($pid3,$outfh3)=pipeline($outfh2,"sort -rn"); warn "sort pid: $pid1\n"; warn "uniq pid: $pid2\n"; warn "sort2 pid: $pid3\n"; while (<$outfh3>) { print; } sub pipeline { my($in,@cmd)=@_; my($out,$child_write); pipe($out,$child_write) or die "pipe error: $!\n"; my $pid = fork; if (!defined($pid)) { die "fork error: $!\n" }; if ($pid) { # Parent close($child_write); return($pid,$out); } else { # child close($out); if (fileno($in) != 0) { close(STDIN) or die "Couldn't close STDIN: $!\n"; POSIX::dup2(fileno($in),0) or die "Couldn't dup to stdin: $!\n"; } close(STDOUT); POSIX::dup2(fileno($child_write),1) or die "Couldn't dup to stdout: $!\n"; exec(@cmd) or die "exec error: $!\n"; } }
      sgifford, that is perfect! This code works regardless of the input to the first app (which resisted killing when it was reading from a net stream). Ignoring $SIG{CHLD} took care of zombies. (strangely, the auto-reap code from perlipc would kill my server process as well)

      This goes in the tool box, for sure. Thanks very much!