in reply to Parallelization of heterogenous (runs itself Fortran executables) code

If your target OS is one with a native fork then that or one of the CPAN wrappers is possibly the way to go, but if you wanted a threads solution, your attempt was making a mountain out of a molehill.

There is certainly no logic to using 3 queues to pass the path, the filename, and the path+filename. Pass it once and split it up there if need be.

And there is no need to count the threads or detect when one has finished. Just start the number your want to use (say 4), and have them loop over the queue. As soon as they finish one piece of work they'll pick up the next automatically. It is also far more efficient than starting a new thread for each file.

When you've finished pushing all the filenames, push 4 undefs and the threads will self terminate when there is nothing left to do. Simplicity itself.

#! perl -slw use strict; use threads; use Thread::Queue; use File::Find; use File::Copy; use File::Path; my $THREADS = 4; my $Q = new Thread::Queue; sub worker { while( my $path = $Q->dequeue ) { print "Creating directory" . $workdirectory . "\n"; mkpath($workdirectory); copyfiles( $workdirectory, $FilenamewithpathQueue->dequeue, $filename[0] ); print "Processing Calculations...\n"; open(RUNSCRIPT,">runscript.bash"); print RUNSCRIPT <<EOD cd $workdirectory perl scriptthatincludesfortran.pl >& $workbasedirectory/$filen +ame[0].log EOD close(RUNSCRIPT); if (system("bash runscript.bash") != 0) { die "failed!\n";} else { print "done!\n"; } print "bash $workdirectory/runscript.bash\n"; print "\n"; } } my @threads = map{ threads->create( \&worker ) } 1 .. $THREADS; find( sub{ $Q->enqueue( $File::Find::name ) }, $basedirectory ); $Q->enqueue( (undef) x $THREADS ); $_->join for @threads;

The code is untested because I couldn't make much sense of your OP code, as half the variables in it are never declared and never initialised.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^2: Parallelization of heterogenous (simple with threads)
by Jochen (Acolyte) on Nov 21, 2007 at 10:09 UTC
    Yes, i know. This is because ist is just the part of my script that I thought would be relevant. Splitting up afterwards would also be my solution. Up to now I was just too lazy to do it. Well, I guess polishing kann be done after it works...
    Thanks anyway. Jochen