Asynchronously capturing output from an external program

Furple has asked for the wisdom of the Perl Monks concerning the following question:

Perl Monks please help!

I'm building integration tests. I have a bunch of machinery that processes an email and then spits out the modified email as a file in a new directory. I've also got a script that generates an email and passes it to the machinery. What I'm trying to do is capture the output from my machinery to figure out the name of the file it's just created.

I'm using Test::More to kick things off.
startup looks like:

sub startup : Test(startup)
{
   system("perl $home/filterd.pl &");
   sleep(3); #give the program some time to set up it's children etc.
}
[download]

Then in a set of individual test I fire:

system("perl $home/filtermail.pl -test -to guy\@whatever.net -id 1234"
+);
[download]

In order to generate results. The idea is that afterwards I check the output from the first program and then process the file it's generated.

Anyone have ideas as to how I capture the output from the first program? The first program uses syslog() to (if run from a terminal) output debug messages etc to the screen. (Presumably STDOUT?) But I can't capture those messages with STDOUT, the program just hangs waiting for input. My other attempt was capturing the results of the last command executed with $! (in the case of the second chunk of code, by running it with `cmd` instead). But this only gets me THAT programs output saying that it's sent information along. In any case I'm stumped.

update: Added a point of clarification and formatting.

Comment on Asynchronously capturing output from an external program Select or Download Code

Replies are listed 'Best First'.
Re: Asynchronously capturing output from an external program by ikegami (Patriarch) on Nov 08, 2010 at 22:36 UTC
`open(my $fh_from_child, '-\|', 'perl', "$home/filtered.pl")` [download] Keep in mind that the child might buffer its output when it's not connected to a terminal. If the reader hang, it could simply mean the child hasn't emptied its buffer yet. Using a pseudo tty (like Expect does) usually convinces the child not to buffer its output. Update: Added paragraph about buffering.	[reply] [d/l]
Re^2: Asynchronously capturing output from an external program by Furple (Novice) on Nov 09, 2010 at 17:55 UTC
File handle worked just fine. ...now I'm trying to get the output back out of it. Now I'm hanging when I try to extract the output from the handle. My code looks like: `print "\n PROCESSING \n\n"; while (<$fh>) { print "$_ \n"; if($_ =~ m/"urlid":"(.*?)"/) { print "found it! \n"; $target = $1; print "terminating \n"; last; } }` [download] I get "Processing" and then the first line from the output before it just hangs. I'm not entirely sure what's going on there. I'll try messing around with Expect and see if I can get that to work.	[reply] [d/l]
Re^3: Asynchronously capturing output from an external program by ikegami (Patriarch) on Nov 09, 2010 at 21:35 UTC
I get "Processing" and then the first line from the output before it just hangs So it's waiting to receive a newline from the child. The child hasn't sent one at all, or it's waiting in a buffer the child hasn't flushed.	[reply]
Re: Asynchronously capturing output from an external program by bingos (Vicar) on Nov 09, 2010 at 11:16 UTC
Capture::Tiny See also Dave Golden's talk on Capturing output The Asynchronously in your heading is misleading as you don't actually appear to want that at all	[reply]
Re^2: Asynchronously capturing output from an external program by Furple (Novice) on Nov 09, 2010 at 17:43 UTC
Mmm. Asynchronous is perhaps not 100% correct but it's not as simple as do one thing, then do something else. I want to fire the main code at one point (filterd), do something(fire filtqmail), and then check the main program's output. I can't fire the code and capture its output and THEN prod it into motion, which at least from an initial read is what it looks like Capture::Tiny does. While it's interesting and probably a useful thing for me to know about in the future I'm not sure that it accomplishes what I need it to. So far the best answer has been a file handle to the output so I can check it when I want to. That said the link on Dave Golden's talk was awesome. Thanks for that.	[reply]
Re: Asynchronously capturing output from an external program by tod222 (Pilgrim) on Nov 09, 2010 at 05:09 UTC
Is there any reason you can't have the command for the first program direct it's STDOUT to a temporary file and then read that temporary file? That's a common way to address this situation.	[reply]
Re^2: Asynchronously capturing output from an external program by Furple (Novice) on Nov 09, 2010 at 17:39 UTC
I'm doing integration tests (as I stated in the second sentence) and so I don't want to have to re-write any of the existing machinery. I only want to check to make sure that it does what it's supposed to do. Otherwise there are a number of less painful ways to do this. I'm not sure why piping the startup command for the first program to a file hadn't occurred to me. Although... as it turns out it looks like that doesn't capture all of it's output... strange.	[reply]