Re^2: Running Parallel Perl Script on Linux Cluster

a similar result can be attained changing the shell wrapper instead of the perl script that runs on the cluster nodes:

#!/usr/bin/bash

cd ~/some_dir
(perl mycode.pl $1 $2 > ~/some_out_dir/param_set$1.out.tmp
  && mv ~/some_out_dir/param_set$1.out)
    || touch ~/some_out_dir/param_set$1.fail
[download]

then, on the master you will have to poll from time to time to see if all the result files exist or if "fail" files are there, to requeue the jobs.

Though, it should be better ways to synchronize partial jobs over the cluster.

Comment on Re^2: Running Parallel Perl Script on Linux Cluster Download Code

Replies are listed 'Best First'.
Re^3: Running Parallel Perl Script on Linux Cluster by monkfan (Curate) on Apr 02, 2006 at 07:11 UTC
...you will have to poll from time to time to see if all the result files exist or if "fail" files are there, to requeue the jobs. Hi salva, Thanks for the answer. Roughly, I suppose that one would use glob to check the file (fail or not)? But not sure how to 'reqeue' the jobs. Can you give a simple example how would one do the above step as you suggested? Regards, Edward	[reply]
Re^4: Running Parallel Perl Script on Linux Cluster by salva (Canon) on Apr 02, 2006 at 14:42 UTC
the best way to check if a file exists is with the `-f` operator (it is documented in perlfunc). To requeue a job, you would need to run the corresponding `qsub` command again, ensure that you delete the "fail" file first with `unlink`.	[reply] [d/l] [select]
Re^4: Running Parallel Perl Script on Linux Cluster by Anonymous Monk on Jun 20, 2007 at 15:09 UTC
Hi monkfan, Can you share how the code worked? I'd like to try out a variation on a 32 proc cluster in a classroom environment. Thanks! vanallp	[reply]