Starting with a script does around a dozen lengthy analysis tasks, so I'm trying to update it to use a cluster and do the jobs in parallel. Each analysis script is submitted with a load sharing facility (LSF) Using LSF to monitor the analysis jobs is frowned upon, as that tends to put a load on LSF and keep it from doing more important things. So, I need to use flag files that tell me when each analysis is done.

This submits a job and waits for completion: 

use strict;
my $jobID = `bsub -q long -o $cl_log -J seqPipe "$jobCmd"`;
print "Waiting for analysis work.\n";

while (1){ 
    last if ( -e "$resultsDir/analysis_completed" );
    print LOG ".";
    sleep 15;
}

print "\nFound completion flag.  Analysis jobs are done.\n";

[download]

Assuming each analysis job creates its own flag to indicate completion, this will get more complicated when I'm trying to monitor a dozen jobs. Later on, there's the possibliity that some jobs will have their own set of analysis jobs that they need to monitor.

I'm thinking this while statement will end up with a set of tests, one for each job. Still, it seems inefficient to keep testing for all those files until the last one has finished. Any ideas on better ways to grow this system?

In reply to Using flag files to monitor cluster jobs by bwelch

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.