fx has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I have a few little perl programs which watch system logfiles (eg. /var/log/maillog) and log certain things into a database. However, these little programs have been known to just suddenly die in the past.

My company is becoming more reliant on these programs (they were initially simple testing tools) and want to monitor them so that they can be restarted if they die. However, they wish to control this from their 'network monitor' machine.

The 'network monitor' sits on a machine and watches various services running on various servers using TCP, UDP or ICMP tests. I have been given the challenge to get my programs to work with this setup.

My current suggestion is to fork/thread/whatever a little TCP or UDP client from my programs which would then respond to the monitoring machine's requests. In people's opinions, how good is this solution?

I suppose my other approach would be to ignore that the 'network monitor' exists and run my own monitoring script which would restart the programs should they die. One way to do this would be to simply watch the output of 'ps', but I'm after a more Perl-based solution. Comments?

fx

Replies are listed 'Best First'.
Re: External monitoring of a Perl program
by blakem (Monsignor) on Sep 18, 2001 at 03:02 UTC
    I would recommend a full fledged network monitoring system that resides on both the client and server... I've used Big Brother in the past, but have been considering a switch to netsaint.

    Both of these systems allow you to write plug-ins (read short scripts) that check on anything you want them to. These plug-ins send status messages to the server, which updates a central repository of status information. The server will then notify you if critical status items are missing or broken.

    -Blake

[reply]
      I agree with blakem. I currently use Big Brother and write external plugins (most in Perl) to report to the central server. You may also want to take a look at Spong, which is a perl-based derivative of Big Brother.

      Dave

[reply]
Re: External monitoring of a Perl program
by traveler (Parson) on Sep 18, 2001 at 03:16 UTC
    If the "monitoring machine" has an SNMP manager, you might use SNMP or Net::SNMP to build a small agent (code that responds to SNMP requests) that responded to queries about status and would stop and/or restart your program. The agent could be part of your existing program or a separate program that starts and stops the real application and can check if it is running. Of course if the agent dies, all beta are off.

    I like the idea of integrating the agent into your program: then you could possibly return "meaningful" information to your "monitoring machine" and possibly even generate SNMP traps if an interesting event occurs. You could then have a program to restart it if it dies*.

    Of course, if the monitoring machine is browser-based, I'd use an agent that works that way, although there is Apache::WebSNMP if you want to talk to an SNMP agent via a web interface...

    * If you fork() a child process, wait blocks until it dies. This is what system does as in nardo's example. On some platforms (e.g. *nix) you can arrange to have a signal sent to your process when a child dies, so you can do something else while the child is doing its work.

    HTH, --traveler

[reply]
[d/l]
[select]
Re: External monitoring of a Perl program
by miyagawa (Chaplain) on Sep 18, 2001 at 04:05 UTC
[reply]
Re: External monitoring of a Perl program
by nardo (Friar) on Sep 18, 2001 at 02:59 UTC
    So long as you can make the programs not fork into the background you can use a simple wrapper like
    #!/bin/sh

while true
do
/path/to/program
done

    [download]
    or if a perl solution is required
    #!/usr/bin/perl

while(1)
 {
   system('/path/to/program');
 }

    [download]
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom