Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris

How to log all output from a program?

by absolut.todd (Monk)
on Oct 15, 2007 at 07:21 UTC ( #644847=perlquestion: print w/replies, xml ) Need Help??

absolut.todd has asked for the wisdom of the Perl Monks concerning the following question:

I am writing an application that uses a large number of custom scripts and CPAN modules together. As part of the requirements for the project, I need to be able to log all the output from the program, both STDOUT and STDERR.

I am using this module that I have written:

package Tie::Annotate; use base qw(Tie::FileHandle::Base); use IO::File; use strict; use warnings; sub TIEHANDLE { my $class = shift; my $annotation = shift; my $handle = new IO::File ">> test.err" or die "Cannot open file: +$!\n"; #$self->{FH} = *$FH; return bless {FH => $handle, ann => $annotation}, $class; } sub PRINT { my $self = shift; my $string = shift; my $handle = $self->{FH}; print $handle "$self->{ann} " . $string; } 1;
And early in the program I declare:
tie *STDOUT, "Tie::Annotate", 'OUT'; tie *STDERR, "Tie::Annotate", 'ERR';

And this generally works correctly, outputting to STDOUT gets appended to the error log with OUT at the start of the line and STDERR gets appended to the error log with ERR and the start of the line.

But there is a problem; one of the CPAN modules (Cvs) relies on IPC::Run which dies when STDERR and STDOUT are captured like this.

FWIW the actual error message is

 Can't call method "slave" on an undefined value at /home/FCS/teh/perl/lib/perl5/site_perl/5.8.5/IPC/ line 2816

IPC::Run's verion is $VERSION = " 0.80";

Is there something crazy with what im trying to do?

A suggestion of a better way to catch STDOUT and STDERR would be appreciated if there is any... Unfortunatly, Log4Perl is not an option.

Replies are listed 'Best First'.
Re: How to log all output from a program?
by shmem (Chancellor) on Oct 15, 2007 at 08:01 UTC
    Is redirecting STDOUT and STDERR of the invoking shell an option?
    (perl -le 'print "foo"; print STDERR "bar"' | sed 's,^,OUT ,' | tee -a + log.stdout ) 2>&1 1>/dev/tty | sed 's,^,ERR ,' | tee -a log.stderr

    Of course that doesn't work if there's no tty.

    update: replace /dev/tty with /dev/null in that case. Or better, if you don't need output to the terminal

    (perl -le 'print "foo"; print STDERR "bar"' | sed 's,^,OUT ,') 2>&1 1> +>log.stdout | sed 's,^,ERR ,' >> log.stderr


    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
      you can do he same from perl:
      my $pid = open STDOUT, '|-'; unless ($pid) { defined $pid or die "unable to fork new process: $!"; open my $tee, '>>', $stdout_log_fn or die "unable to open '$stdout_log_fn'"; select $tee; $| = 1; select STDOUT; $| = 1; while (<>) { print $tee "OUT $_"; print $_; } exit(0); } # and repeat for STDERR
        Ah, of course. But you also can do that from the outside :-)

        BTW, if you open/close the filehandle inside the loop (and perhaps include $$ in the output) it is less probable that multiple processes mess up the logfiles.


        _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                      /\_¯/(q    /
        ----------------------------  \__(m.====·.(_("always off the crowd"))."·
        ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
        BTW, I have just released to CPAN File-Tee that's an augmented version of the code above.
Re: How to log all output from a program?
by bart (Canon) on Oct 15, 2007 at 10:37 UTC
    I have no neat solution for you, just some general remarks...

    Tieing filehandles when calling external programs just doesn't work. I bet that is why IPC::Run dies.

    What does work for external programs, is redirecting these filehandles to files, or to real filehandles. Child programs will then inherit these filehandles. For example:

    open STDOUT, ">stdout.txt"; open STDERR, ">stderr.txt"; system($commandline);
    Now the command will send its output to stdout.txt and stderr.txt respectively.

    So, what to do...? I'm thinking of using an intermediate program/script, which captures the output from your external program (perhaps with IPC::Open3), reformats it, and prints it out, formatted, to the real log file.

Re: How to log all output from a program?
by Corion (Patriarch) on Oct 15, 2007 at 13:52 UTC

    By weird coincidence, clkao just released IPC::Run::SafeHandles, which claims to handle your exact situation better. Maybe this gets merged back into IPC::Run some day but for the meantime, this might be a stopgap measure - the magic seems to get activated when $ENV{FCGI_ROLE} is set, but a look at the code might prove more instructive.

Re: How to log all output from a program?
by DrHyde (Prior) on Oct 15, 2007 at 11:37 UTC
    Hrrm, the problem with IPC::Run might explain why I can't properly test it using CPAN::Reporter - thanks!
Re: How to log all output from a program?
by dokkeldepper (Friar) on Oct 15, 2007 at 08:26 UTC
    To widen the perspective on your logging problem: log4perl.
      He specifically said log4perl wasn't an option as the very last sentence :)

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://644847]
Approved by Corion
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (5)
As of 2023-03-31 09:26 GMT
Find Nodes?
    Voting Booth?
    Which type of climate do you prefer to live in?

    Results (75 votes). Check out past polls.