I'm aware of this; that's why I was surprised when I compared the runtimes of the one worker thread vs. two worker threads runs. The processing time was approx 43 minutes for both cases.

I looked at the output of ps -u while my program was running: in the two threaded case there were two R processes running, both using 99% CPU power, and from the output it seemed that they were dividing the jobs among themselves. Yet, the processing took as much time as in the single threaded case.

As for the question about messages between threads: The manager thread uses Storable to create work units from the dataset arrays; the frozen arrays are placed on the input queue. These are fairly large.
The work threads use a second queue to get the results back to the manager thread; however, the results are just hashes with about a dozen keys.

I'll construct a minimal example and get back to you.

In reply to Re^4: Using threads to run multiple external processes at the same time by kikuchiyo
in thread Using threads to run multiple external processes at the same time by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.