in reply to Using threads to run multiple external processes at the same time

I guess the obvious answer is "threads",
Actually, the obvious answer is "fork".
  • Comment on Re: Using threads to run multiple external processes at the same time

Replies are listed 'Best First'.
Re^2: Using threads to run multiple external processes at the same time
by Sewi (Friar) on Aug 31, 2009 at 21:39 UTC
    I'm a fan of fork, I use it much more than threads, but this is a better job for threads if running on a single computer. Thread-queues are much faster and very much easier as shared memory. The same is true for thread control vs. forked-child control (like waiting and not missing childs).
[reply]
      The OP wants to go to a situation where different computers are used. That's never going to work with threads. You'd better off starting with forks and doing IPC with sockets than first implementing a thread based solution, and then redoing part of the work if you take it to the next level.

      Besides, I don't trust that every OS treats threads in such a way that it's actually running different threads of the same process on different CPUs. Or that it gives 2 threads the same number of run-slots as it does with 2 processes.

[reply]
        The OP wants to go to a situation where different computers are used. That's never going to work with threads.

        Look again! The OP want's to service multiple concurrent processes, by feeding them unique subsets of a large common dataset. Those external processes cannot make use of COW shared data, it has to be fed to them via a file or pipe. All that is required to extend the threaded solution offered to use multiple machines, is to substitute rsh or ssh for open2.

        The OPs program starts one thread to perform the subsetting and feeding of that subset to an R instance. No messy interleaving of reading and writing to multiple sources and targets. Each thread performs the subsetting; starts the R instance; writes the data; reads the results; shuts down the process (or feeds it another). Whether that R instance runs locally or remotely is entirely transparent to the threading

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom