in reply to Re^2: Using threads to run multiple external processes at the same time
in thread Using threads to run multiple external processes at the same time

The OP wants to go to a situation where different computers are used. That's never going to work with threads. You'd better off starting with forks and doing IPC with sockets than first implementing a thread based solution, and then redoing part of the work if you take it to the next level.

Besides, I don't trust that every OS treats threads in such a way that it's actually running different threads of the same process on different CPUs. Or that it gives 2 threads the same number of run-slots as it does with 2 processes.

  • Comment on Re^3: Using threads to run multiple external processes at the same time

Replies are listed 'Best First'.
Re^4: Using threads to run multiple external processes at the same time
by BrowserUk (Patriarch) on Sep 01, 2009 at 01:38 UTC
    The OP wants to go to a situation where different computers are used. That's never going to work with threads.

    Look again! The OP want's to service multiple concurrent processes, by feeding them unique subsets of a large common dataset. Those external processes cannot make use of COW shared data, it has to be fed to them via a file or pipe. All that is required to extend the threaded solution offered to use multiple machines, is to substitute rsh or ssh for open2.

    The OPs program starts one thread to perform the subsetting and feeding of that subset to an R instance. No messy interleaving of reading and writing to multiple sources and targets. Each thread performs the subsetting; starts the R instance; writes the data; reads the results; shuts down the process (or feeds it another). Whether that R instance runs locally or remotely is entirely transparent to the threading

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom