If you need a quick hack, I can offer you two low level solutions (sort of). Assuming you are using some sort of unix, you might want to try this:

1. Use a wrapper script to start your program. It can restart it whenever the program crashed:

#!/bin/sh
while true
do
  start_your_program
  sleep xxx
done
[download]

2. You might grep through the process list to see if you program is running. This works if it has a sufficiently long and distinct name. Try on a shell:

ps aux | grep your_program_name | grep -v grep | wc -l
[download]

ps -ef | grep your_program_name | grep -v grep | wc -l
[download]

Hope that's a bit useful. Kind of old techniques, but it still works (most of the time) ;-)

Andreas

Anyway, you could modify your script so that it saves it's own PID in a file. (You can get the PID from the special variable $$). Then have your new script (executed from cron every 5 minutes or so) look in that file and then check if that PID is active. If the PID is running, be sure to check the args too, so it's really your script that has that PID.

This would be really easy in Unix. I guess it could be done in NT too? I'll let someone else answer that...

Secondly, be sure to log all the starts and restarts or at least notify somebody about them, so that you always have a good idea of what is going on. Otherwise, you are likely to forget about it after it "just works" (i do this all the time too, its an easy habit to get into) and never be aware of what is going on.

Good luck,
Mark Logan

Good point! Both of them. That would make a not-so-quick but better hack. Append a timestamp and a start note / end note with exit code to a log file, to have a history. You might also send an email in case of a crash. Let's see ...

#!/bin/sh
while true
do
  start_your_program
  RC=$?          # return code
  if [ $RC -gt 0 ]
  then
    date >> /var/log/your_log_file
    echo "ABNORMAL program termination, rc = $RC" >> /var/log/your_log
+_file
    echo "terminated at `date`, restarting | mail -s "Problem with zzz
+" root
    sleep xxx
  else
    date >> /var/log/your_log_file
    echo "Normal program termination, rc = $RC" >> /var/log/your_log_f
+ile
    sleep yyy
  fi
done
[download]

You can do further checking on the exit code as Mark suggested and act diffently according to the exit code (see case statement in your shell manual).

Andreas

#!/usr/bin/perl

use strict;

# Loop indefinitely
while (1)
{
    # Fork off a child process
    my $kidpid = fork();
    if ($kidpid)
    {
        # This is the parent process.  Wait for
        # the child to exit
        waitpid($kidpid, 0);
        
        # Put some code here to send you an alert
        # when the child dies.  You can also
        # check the child's exit condition here
        # with $?.
    } elsif (defined($kidpid)) {
        # This is the child process.
        # Put your original script in here
        # or just use exec()
    } else {
        die "could not fork";
    }
}
[download]

That will fork off a child process to handle your script and restart it every time it dies

-Matt

If you are on WinNT I would suggest that you make your program a service.

#!/usr/bin/perl -w

use strict;

launch_child();
$SIG{CHLD}=\&launch_child;
sleep 60 while(1);


sub launch_child{
  print "$$ parent spawning a child\n";
  my $pid=fork;
  if(!$pid){
    print "$$ inside the child\n";
    # all the code that does the real work is in here
    # all the other stuff is just a wrapper to keep
    # this bit going
    sleep 60 while(1);
  }
}
[download]

That's an interesting solution, but it's likely to dump core. From perlipc:

Do as little as you possibly can in your [signal] handler ... because on most systems, libraries are not re-entrant; particularly, memory allocation and I/O routines are not. That means that doing nearly anything in your handler could in theory trigger a memory fault and subsequent core dump.

-Matt

Under normal situations that would be the case, but since my main program doesn't do anything other than sleep there are no non-reentrant pieces of code that could be interrupted by the dying process (causing a core dump). I have used this technique before and it has proven to be quite stable.

mikfire

Because you are in trouble if your script keeps crashing all the time. This would generate a severe load on your machine. At least, that would keep me from hooking it up into the systems runlevel until I am _very_ sure this program behaves well even in insane situations.

Just my two cents ;-) ...

Andreas

That problem already exists. If the script is dying a lot, it doesn't matter what respawn mechanism is used. There will still be a significant load placed on the system.

The problem description, though, makes it sound like it only happens occassionally, which has made it hard to debug. In that case, I would rather let a proven and well-known mechanism like initd do the monitoring for me than having to debug both the respawn code and the code that is dying in the first place.

Second, initd is usually pretty smart and stops trying a job if it is respawning too quickly. So your load increase for a bit, but initd does the Right Thing and stops it from becoming a fork bomb.

mikfire

Most versions of init that I've run into will notice that an entry has restarted over and over in a short time and just complain to the console that entry "xyz" is restarting too much and it won't restart it anymore until you change the inittab.

You see this a lot with flaky terminals (people still know what a terminal is, don't they?) where getty keeps croaking and init just gives up on it.

I think init is perfect solution for this problem if you have access to it.

$SIG{PIPE} = sub { #handle error }
[download]