Well, I was waiting around and working on another part of the code when it hit me. I am still not sure how to do what I want the "right" way (assuming there is such a thing). But I came up with what I think is a pretty reasonable solution:

I redirected STDOUT and STDERR to a file when I run the application. Once the file is created, I use File::Tail to watch the file for the telltale string letting me the database is ready to accept connections and break out of the while loop that File::Tail is conveniently allowing me to use to block with. Here is the working code if anyone happens to ever have a similar problem

Or maybe I am the only knuckle dragger that has never run into this kind of issue before and solved it a long time ago?

sub cge_start {
  my($arg_ref) = @_;

  #=+ iterate over provided config and replace defaults
  while(my($k,$v) = each %$arg_ref) {
    #=+ Replace each default value if the key exists.  Perhaps paranoi
+d, but prevent arbitrary/unexpected settings
    $arguments{$k} = $v if exists $arguments{$k};
  }

  #=+ Make sure we have a bare minimum of settings
  return undef if $arguments{dataDir} eq '' || !-d $arguments{dataDir}
+ || $arguments{imagesPerNode} eq '' || $arguments{nodeCount} eq '';

  #=+ Create the results directory
  #   make sure that our directory has a trailing slash "/"
  $arguments{dataDir} .= '/' unless $arguments{dataDir} =~ m/\/$/;

  #=+ Create the results directory if it does not exist
  mkdir $arguments{dataDir}.'results', 0744 if !-d $arguments{dataDir}
+.'results';
  $arguments{resultDir} = $arguments{dataDir}.'results';

  #=+ Create a temporary directory to store all the junk files we crea
+te for easy cleanup later
  mkdir $arguments{dataDir}.'temp', 0744 if !-d $arguments{dataDir}.'t
+emp';

  #=+ Create the logfile string (does not need to exist)
  $arguments{logFile} = $arguments{dataDir}.'cge_logfile.log';

  #=+ Make sure we have both the number of instances and number of des
+ired nodes
  return (undef,undef) unless $arguments{imagesPerNode} > 0 && $argume
+nts{nodeCount} > 0;

  #=+ Create a cleanup script that we will use to kill "this" server s
+ession
  $arguments{cleanupScript} = $arguments{dataDir}.'temp/cleanup.sh';

  #=+ Make sure we have a free TCP port to use
  my $port = 3750;
  if(check_port($port)) {
    $port++;
    while(check_port($port)) {
      $port++;
    }
  }
  $arguments{queryPort} = $port;

  #=+ Concatenate all the non-blank command line arguments
  my @args;
  while(my($k,$v) = each %arguments) {
    push @args, '--'.$k.' '.$v.' ' unless $v eq '';
  }
  my $arg_string = join('',@args);

  #=+ Run the launcher
  #exec($cge_launch.' '.$arg_string.'> /dev/null 2>&1 &');
  my $success = run(command => $cge_launch.' '.$arg_string.' > '.$argu
+ments{dataDir}.'temp/stdout.txt 2>&1 &', verbose => 0);

  #=+ Something went wrong, so just return undef
  return (undef,undef) unless $success;

  my $file = File::Tail->new($arguments{dataDir}.'temp/stdout.txt');
  while(my $line = $file->read) {
    last if $line =~ m/Starting port forwarding/;
  }

  open(my $CU,'<',$arguments{cleanupScript}) or croak $!;
  my $text = <$CU>;
  close($CU);

  my $pid;
  if($text =~ m/scancel (\d+)/m) {
    $pid = $1;
  }
  return ($pid,$port);
}
[download]

If you are waiting for some file to change, then some variant of inotify will tell you the file changed without looping continuously in a tail loop burning Mips to not much effect. You can also get notified if a directory changes, some new file appears, perhaps some long awaited "I am finished and ready for business" file.

Update: I suppose also if you opened the file for read, then while(<FILE>)... will cause a program pause in the OS waiting for the next line to appear. You can examine each line waiting for the right msg to be written. Of course this could "deadlock" if the if the other program hangs. But if it dies and file closes, then you should get an EOF. The basic rationale behind these 2 ideas is not burn a lot of Mips waiting for something that itself is burning a lot of Mips where your "watcher" slows things down. I haven't tested, but I don't think you get an EOF if somebody else has the file open for "write". I have used inotify before and that will work.

Update: eatingmyownwords = I now think that that this won't work unless you are using some sort of trick like Tee and reading from a pipe which is left open. I think you get EOF when you reach the current extent of text in the file whether it is open or not for additional writes - that's what tail does.

So, are you saying my File::Tail approach is OK? Or is there some way that is better? I'm totally open to doing in whichever way is best...the approach I took was simply to "git 'er done"...certainly not because I was positive it was the most elegant solution...

Hmm, your first question sounded like you were talking about shell job control (background and foreground process groups), but it seems you're just spawning a process and want to read its output. Looking at your code, I don't see any particular reason to use File::Tail... you could've just opened temp/stdout.txt as usual (with the open function) and read it line by line? Anyway, people normally use FIFOs for that, you can make one with POSIX::mkfifo, perlipc has an example.

, I don't see any particular reason to use File::Tail

oh, I see, the other process is constantly appending to the file and you don't want to get EOF.