submitting job to a queue

mjr1n1 has asked for the wisdom of the Perl Monks concerning the following question:

hello monks..

I am trying to submit a job (which is embedded in a shell script) using 'qsub'. I tried running the code using various options in perl, but each time when it runs, it submits the job but exits the perl program and does not wait for the job to be finished.

I need the program to wait till the queued job is finished and then based on job output do other things later. Any help will be appreciated. I tried..

$job = "/usr/bin/qsub pbs_job.sh";

$status = system($job);    #it submits the job here and
                           #exits perl code

If ($status == 0) {

do something

}
[download]

2. using pipes

$job = "/usr/bin/qsub pbs_job.sh";

open(my $status, "-|", $job)
   or die "couldn't launch qsub job: $!/$?\n";

for (<$status>) {
       print "$_";
}
close $status;
[download]

thanks.. mjr.

Comment on submitting job to a queue Select or Download Code

Replies are listed 'Best First'.
Re: submitting job to a queue by pjotrik (Friar) on Jul 10, 2008 at 14:08 UTC
Queueing is asynchronous by nature, qsub will only submit the job. I see several possible solutions: - Check `qstat` periodically, until it says the job is finished - Use some synchronization tool, e.g. hang your controlling script on a semaphore and change your queued script to raise this semaphore when it's finished - when submitting the job, tell the batch server to mail you when job's status changes, handle the incoming mail by some script. That's probably too complicated for this case	[reply] [d/l]
Re: submitting job to a queue by toolic (Bishop) on Jul 10, 2008 at 14:07 UTC
Do you mean that your perl script actually quits when system is called, and you never reach your "if" statement? If the qsub you are referring to is the SGE submission command, then I can think of a couple of potential solutions. I do not have much experience with SGE, but when I google qsub, and search for "wait", I see the "-sync" option, which states: `-sync y[es]\|n[o] Available for qsub. -sync y causes qsub to wait for the job to complete before exiting. ...` [download] Another possibility is to submit the job as you are doing, then wait for the job to finish by polling its status using qstat.	[reply] [d/l]
Re^2: submitting job to a queue by mjr1n1 (Initiate) on Jul 10, 2008 at 14:31 UTC
No, qsub submits the job correctly and the job starts as well and runs in the background. But, the moment job gets submitted and before the job is finished, it comes back to the next statement in perl (e.g. "if" statement) and executes it and then exits the perl program. I need the perl program to just wait after submitting the qsub job and till it is finished, and then go to the next statement (i.e. "if" statement) thanks..	[reply]
Re^3: submitting job to a queue by Anonymous Monk on Jul 08, 2018 at 06:36 UTC
I know this was posted a long time ago but I came across the same problem and in case it might help someone else, I solved it by using the "-W block=true" qsub option for PBS systems(i think the problem arises because "the qsub command exits after returning the ID of the new job" and not after completing the task)	[reply]
Re: submitting job to a queue by baxy77bax (Deacon) on Jan 19, 2009 at 08:49 UTC
i know it's a bit late and you probably solved the problem but here is an extra solution if the output of the job is some kind of a file or a bunch of files that you cp in some directory you can check if all the files are there and then continue: `my $qsub = qx/qsub -pe mpi 5 -t 1:$filecount:2 $bash/; print "$qsub\n"; # check for sge to finish my $r; my $fileoutcount = $filecount + 2; while($r < $fileoutcount){ opendir(DIR, "./out/BLAST/out") \|\| die "$!"; $r = 0; foreach (readdir DIR){ $r++; } closedir DIR; sleep 500; }` [download] this was my solution, running the job on sge by bash script. and now i'm looking for a more direct approach, and SGE module looks pretty ok for now. baxy	[reply] [d/l]