Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Multiplexing STDOUT and STDERR

by kilinrax (Deacon)
on Nov 12, 2003 at 18:56 UTC ( #306592=perlquestion: print w/replies, xml ) Need Help??

kilinrax has asked for the wisdom of the Perl Monks concerning the following question:

I'm currently working on a wrapper script which needs to execute a script, and do the following:

  • Append STDOUT and STDERR to two separate log files.
  • Still echo STDOUT and STDERR to the terminal.
  • Store STDOUT and STDERR in variables / temp files / whatever.
  • Send a warning email if the script returns an error code, including the stored STDOUT and STDERR.

Previously, the script was written in shell, and accomplished the first two requirements thusly:

( ( perl | tee -a output.log ) 3>&1 1>&2 2>&3 3>&- | tee -a er +ror.log ) 3>&1 1>&2 2>&3 3>&-

However, capturing the return code of the script as well, even if it is possible, is beyond my script-fu.
Given that, and the fact that writing a wrapper to a Perl script in shell strikes me as inherently nasty, I set about recoding it in Perl.

The obvious solution seemed to be to redirect STDERR and STDOUT, and then set @ARGV and 'do' the script.
I could use IO::Tee to tee between STDOUT or STDERR, an IO::File object for the logfile, and an IO::String object for the variable store (to potentially be used in an email); and to assign the whole lot back to the relevant glob.
i.e. something like:

my ($output_cache, $error_cache); *STDOUT = multiplex( \*STDOUT, $output_cache, "$log_dir/output.$0" ); *STDERR = multiplex( \*STDERR, $error_cache, "$log_dir/errors.$0" ); do $script or die "Could not open '$script': $!" exit; sub multiplex { my ($handle, $cache, $log) = @_; my $cache_handle = IO::String->new( \$cache ); my $log_handle = IO::File->new( $log, 'w' ) or die "Could not open l +og file '$log': $!"; my $tee = IO::Tee->new( $handle, $cache_handle, $log_handle ); return $tee; }

However, this particular approach generates errors of the ilk 'print() on unopened filehandle STDOUT'.
I assume that assigning to globs only works with IO::File objects because they're blessed globrefs internally (though IO::Tee also uses Symbol::gensym in it's constructor, so perhaps I am wrong).

Another obvious approach is using 'select', which works fine for STDOUT, but not for STDERR.

So, my question boils down to two things:

  • Can you see anything obviously wrong in my implementation?
    Is there a trick to assigning to STDERR I've missed?
  • Can you see anything obviously wrong in my design?
    Is it the right way to do things, or even a sensible way? Should I be running the script as a subprocess and filtering the output, rather than altering its output handles? How would you tackle this problem yourself?

Bear in mind that the script I'm wrapping ships with a proprietary software distribution, which we upgrade semi-regularly; so I want to avoid treating it as anything other than a black box, despite the fact that I can see the source.

Replies are listed 'Best First'.
Re: Multiplexing STDOUT and STDERR
by jmanning2k (Pilgrim) on Nov 12, 2003 at 20:42 UTC
      OK, I realize Log4Perl is a lot of work rewriting your scripts, but trust me, it's worth it. For a quick start, follow the easy_init directions, and just replace all your 'print "' with $log->info(), and all the warn or 'print STDERR' calls with $log->warn().

      But, of course you would like to use as much of the existing script as possible. So, to address your current problem, try using \*STDOUT directly in the IO::Tee call. I've found odd behavior when I pass it as another variable, vs passing it as \*STDOUT and \*STDERR. Call multiplex with some sort of indicator instead of the reference (a string perhaps "STDOUT" or "STDERR"), and make separate calls to IO::Tee for each of these.

      To debug this, try using the handles method of IO::Tee. Add a print join(' ', $tee->handles), "\n"; (taken from the perldocs) to your multiplex method to see what it's really producing. Compare the results of using the Handle reference, vs using \*STDOUT.

      I know the above works, but making separate calls to IO::Tee is pretty ugly. This is untested, but you might try creating a new filehandle object pointing to STDOUT. Perhaps something like IO::File->new(\*STDOUT), or more basic, IO::Handle->new_from_fd(fileno(STDOUT),"w"). Pass the return value from one of these to multiplex instead of the reference to the glob.


Re: Multiplexing STDOUT and STDERR
by hardburn (Abbot) on Nov 12, 2003 at 21:02 UTC


    I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
    -- Schemer

    : () { :|:& };:

    Note: All code is untested, unless otherwise stated

Re: Multiplexing STDOUT and STDERR
by demerphq (Chancellor) on Nov 13, 2003 at 00:00 UTC

    Its interesting, but I have a module that I wrote for work that does pretty close to exactly what you ask for here. The job is indeed trickier than it looks. The approach I took was to tie duped STDOUT and STDERR filehandles, and then intercept warnings and dies through a SIG handler (which can be problematic according to the docs, but not so far for me.) This works with the majority of code, but things that also play games with these filehandles can mess things up. Thankfully they seem to be rare.

    Incidentally I too started with IO::Tee, and it worked (with added sig handlers), but as I started adding hooks and dealing with issues ($SIG{__DIE__} will be called even inside an eval is an example, albeit not that good a one) it soon became easier not to bother.



      First they ignore you, then they laugh at you, then they fight you, then you win.
      -- Gandhi

Re: Multiplexing STDOUT and STDERR
by Anonymous Monk on Nov 13, 2003 at 18:26 UTC
    #!/usr/bin/perl use strict; use Symbol; use IO::Select; use IPC::Open3; my $outlog = shift; my $errlog = shift; my $cmd = shift; my @args = @ARGV; open OUTLOG, ">>$outlog" or die "open $outlog: $!"; open ERRLOG, ">>$errlog" or die "open $errlog: $!"; my ( $rdr, $wtr, $err ); my $pid = open3( $wtr, $rdr, $err = Symbol::gensym, $cmd, @args ); close $wtr; my $select = IO::Select->new( $rdr, $err ); my $outstr = ''; my $errstr = ''; while ( $select->handles ) { for my $handle ( $select->can_read ) { my $bytes = sysread $handle, my ($str), 1024; die "sysread: $!" unless defined $bytes; $select->remove($handle), next unless $bytes; if ( fileno($handle) == fileno($rdr) ) { print STDOUT $str; print OUTLOG $str; $outstr .= $str; } elsif ( fileno($handle) == fileno($err) ) { print STDERR $str; print ERRLOG $str; $errstr .= $str; } } } close $rdr; close $err; close OUTLOG; close ERRLOG; waitpid $pid, 0; my $returncode = $? >> 8; my $signal = $? & 127; my $coredump = $? & 128; exit unless $returncode; my $time = localtime; my $mail = \*STDERR; # new Mail::Mailer; print $mail <<END; Process $cmd @args returned $returncode (signal $signal, coredump $coredump) at $time standard output: ==== $outstr ==== standard error: ==== $errstr ==== END close $mail;

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://306592]
Approved by Paladin
Front-paged by davido
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (2)
As of 2022-01-22 17:30 GMT
Find Nodes?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:

    Results (63 votes). Check out past polls.