in reply to Parallelization of heterogenous (runs itself Fortran executables) code

Dictum Ne Agas: “ Do not do a thing already done.

If you're doing a CPU-intensive task, then “threads” won't help you: threads divide the computer's time; they do not multiply it. Only a cluster of actual discrete CPUs will do the trick.

And since “that's tricky,” ... “It's a bird! No, it's a plane! No, it's CPAN To The Rescue!” There are numerous tested-and-debugged modules available to do what you have in mind. (There are also full-fledged batch scheduling systems.) So... enter the mindset that you are not setting out to “build” your solution, but merely to “find” it. Rocket-scientists all over the globe have addressed exactly the same problem, solved it thoroughly, and given away their solution.

Replies are listed 'Best First'.
Re^2: Parallelization of heterogenous (runs itself Fortran executables) code
by Jochen (Acolyte) on Nov 20, 2007 at 22:24 UTC
    "Threads divide the computer's time; they do not multiply it" - AFAIK not if its an smp-machine that would otherwise not take advantage of the extra cores.
[reply]
      It is all moot if one (or more) of the threads is hogging I/O.

      Update 1: while I make a point not to comment on downmodded posts of mine, I have to say that i/o contention among threads is a SERIOUS issue - even on an SMP machine that has multiple cores/cpus. So put that in your pipe and smoke it.

      Update 2: I've been asked by BrowserUk to clarify, so here we go. As it was pointed out, threading simply divides time among related processes.

      On a single CPU system, the issue of a thread hogging i/o is indistinguishable from a simple blocking thread - sibling threads simply won't get a chance to execute until the thread that is executing finishes with its reads/writes.

      On shared memory (multi-core, SMP), the situation changes. Multiple threads may be executing, but if some small subset of the threads are engaging in heavy i/o, things will come screeching to a crawl. My point is that there is an additional contention for i/o bandwidth, and unless there is a high performance disk subsystem available to your threads, simply thowing more CPUs at it won't make a lick of difference - and could in fact make your performance a lot worse than just running the code serially.

      We don't know what the OP's code is doing. If it is reading and writing a lot of data, this could have a significant impact on the performance of the threads.
[reply]
[reply]

In Section Seekers of Perl Wisdom