How to log all output from a program?

absolut.todd has asked for the wisdom of the Perl Monks concerning the following question:

I am writing an application that uses a large number of custom scripts and CPAN modules together. As part of the requirements for the project, I need to be able to log all the output from the program, both STDOUT and STDERR.

I am using this module that I have written:

package Tie::Annotate;

use base qw(Tie::FileHandle::Base);
use IO::File;
use strict;
use warnings;

sub TIEHANDLE {
    my $class = shift;
    my $annotation = shift;
    my $handle = new IO::File ">> test.err" or die "Cannot open file: 
+$!\n";
    #$self->{FH} = *$FH;
    return bless {FH => $handle, ann => $annotation}, $class;
}

sub PRINT {
    my $self = shift;
    my $string = shift;
    my $handle = $self->{FH};
    print $handle "$self->{ann} " . $string;
}

1;
[download]

And early in the program I declare:

tie *STDOUT, "Tie::Annotate", 'OUT';
tie *STDERR, "Tie::Annotate", 'ERR';
[download]

And this generally works correctly, outputting to STDOUT gets appended to the error log with OUT at the start of the line and STDERR gets appended to the error log with ERR and the start of the line.

But there is a problem; one of the CPAN modules (Cvs) relies on IPC::Run which dies when STDERR and STDOUT are captured like this.

FWIW the actual error message is

Can't call method "slave" on an undefined value at /home/FCS/teh/perl/lib/perl5/site_perl/5.8.5/IPC/Run.pm line 2816

IPC::Run's verion is $VERSION = " 0.80";

Is there something crazy with what im trying to do?

A suggestion of a better way to catch STDOUT and STDERR would be appreciated if there is any... Unfortunatly, Log4Perl is not an option.

Comment on How to log all output from a program? Select or Download Code

Replies are listed 'Best First'.
Re: How to log all output from a program? by shmem (Chancellor) on Oct 15, 2007 at 08:01 UTC
Is redirecting STDOUT and STDERR of the invoking shell an option? `(perl -le 'print "foo"; print STDERR "bar"' \| sed 's,^,OUT ,' \| tee -a + log.stdout ) 2>&1 1>/dev/tty \| sed 's,^,ERR ,' \| tee -a log.stderr` [download] Of course that doesn't work if there's no tty. update: replace /dev/tty with /dev/null in that case. Or better, if you don't need output to the terminal `(perl -le 'print "foo"; print STDERR "bar"' \| sed 's,^,OUT ,') 2>&1 1> +>log.stdout \| sed 's,^,ERR ,' >> log.stderr` [download] --shmem _($_=" "x(1<<5)."?\n".q·/)Oo. G°\ / /\_¯/(q / ---------------------------- \__(m.====·.(_("always off the crowd"))."· ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}	[reply] [d/l] [select]
Re^2: How to log all output from a program? by salva (Canon) on Oct 15, 2007 at 08:25 UTC
you can do he same from perl: `my $pid = open STDOUT, '\|-'; unless ($pid) { defined $pid or die "unable to fork new process: $!"; open my $tee, '>>', $stdout_log_fn or die "unable to open '$stdout_log_fn'"; select $tee; $\| = 1; select STDOUT; $\| = 1; while (<>) { print $tee "OUT $_"; print $_; } exit(0); } # and repeat for STDERR` [download]	[reply] [d/l]
Re^3: How to log all output from a program? by shmem (Chancellor) on Oct 15, 2007 at 08:50 UTC
Ah, of course. But you also can do that from the outside :-) BTW, if you open/close the filehandle inside the loop (and perhaps include $$ in the output) it is less probable that multiple processes mess up the logfiles. --shmem _($_=" "x(1<<5)."?\n".q·/)Oo. G°\ / /\_¯/(q / ---------------------------- \__(m.====·.(_("always off the crowd"))."· ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}	[reply]
Re^4: How to log all output from a program? by salva (Canon) on Oct 15, 2007 at 09:10 UTC
Re^3: How to log all output from a program? by salva (Canon) on Oct 16, 2007 at 21:42 UTC
BTW, I have just released to CPAN File-Tee that's an augmented version of the code above.	[reply]
Re: How to log all output from a program? by bart (Canon) on Oct 15, 2007 at 10:37 UTC
I have no neat solution for you, just some general remarks... Tieing filehandles when calling external programs just doesn't work. I bet that is why IPC::Run dies. What does work for external programs, is redirecting these filehandles to files, or to real filehandles. Child programs will then inherit these filehandles. For example: `open STDOUT, ">stdout.txt"; open STDERR, ">stderr.txt"; system($commandline);` [download] Now the command will send its output to stdout.txt and stderr.txt respectively. So, what to do...? I'm thinking of using an intermediate program/script, which captures the output from your external program (perhaps with IPC::Open3), reformats it, and prints it out, formatted, to the real log file.	[reply] [d/l]
Re: How to log all output from a program? by Corion (Patriarch) on Oct 15, 2007 at 13:52 UTC
By weird coincidence, clkao just released IPC::Run::SafeHandles, which claims to handle your exact situation better. Maybe this gets merged back into IPC::Run some day but for the meantime, this might be a stopgap measure - the magic seems to get activated when `$ENV{FCGI_ROLE}` is set, but a look at the code might prove more instructive.	[reply] [d/l]
Re: How to log all output from a program? by DrHyde (Prior) on Oct 15, 2007 at 11:37 UTC
Hrrm, the problem with IPC::Run might explain why I can't properly test it using CPAN::Reporter - thanks!	[reply]
Re: How to log all output from a program? by dokkeldepper (Friar) on Oct 15, 2007 at 08:26 UTC
To widen the perspective on your logging problem: log4perl.	[reply]
Re^2: How to log all output from a program? by deorth (Scribe) on Oct 15, 2007 at 09:48 UTC
He specifically said log4perl wasn't an option as the very last sentence :)	[reply]


laziness, impatience, and hubris
	PerlMonks