I think that I've pretty much confirmed that R does some serialisation of resource usage across concurrent process instances. Even though the evidence is not as clear cut as I would like. In the following console trace, I start 1, 2, 3 & 4 concurrent copies of R performing a ~60 second calculation and timing it with its own built-in timer. I set the affinity of each process to 1 of my cpus to prevent cpu thrash:

for /l %i in (1,1,1) do @start /b /affinity %i rscript 
-e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n
+row=2))})"

   user  system elapsed
  62.24    0.00   62.68

for /l %i in (1,1,2) do @start /b /affinity %i rscript 
-e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n
+row=2))})"

   user  system elapsed
  65.49    0.00   65.60
   user  system elapsed
  65.19    0.01   66.13

for /l %i in (1,1,3) do @start /b /affinity %i rscript 
-e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n
+row=2))})"

   user  system elapsed
  65.61    0.06   65.94
   user  system elapsed
  65.75    0.03   98.98
   user  system elapsed
  65.55    0.00   99.30

for /l %i in (1,1,4) do @start /b /affinity %i rscript 
-e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n
+row=2))})"

   user  system elapsed
  68.83    0.00   69.81
   user  system elapsed
  70.59    0.00   72.71
   user  system elapsed
  67.30    0.03  101.99
   user  system elapsed
  67.22    0.00  102.65
[download]

1. For 1 copy, it maxes out the appropriate cpu for ~62 seconds, and the elapsed time closely reflects the cpu time used.
2. For 2 copies, it almost maxes out the two cpus, but both processes show an ~5% 'concurrency overhead'.
3. With 3 copies, again 2 cpus are maxed, but the third show a less than 50% duty until the first completes at which point it also (almost) maxes.

   The cpu times of the 3 processes all show the ~5% concurrency overhead--probably indicative of some internal polling for resource--but the elapsed times show much greater overhead--nearly 70%.

4. Once we get to 4 copies, the activity traces show 2 maxed and 2 well below 50% until one completes, at which point one of the other two picks up. And same again once the second completes.

   That pretty much nails it (for me) that there is some one-at-a-time resource internal to R that concurrent processes compete for. And the more there are competing, the greater the cost of that competition.

   All of which probably reflects Rs long history and its gestation in the days before multi-tasking concurrent cpu-bound processes was a realistic option.

Note: It could be that the shared resource is something simple like the configuration file or history file or similar; and that with the right command line options to disable the use of those, the overhead can be avoided. I haven't explored this. It might be better to ask the guys that know rather than testing random thoughts.

Whilst looking around I did come across Rmpi which might be a better approach to your problem. Though it appears to be aimed at spreading the load of single calculations over multiple cpus or boxes, rather than running multiple concurrent calculations. You'd need to read a lot more about than I've bothered with :)

If anything, this only served to increase my confusion.

I installed R on my home E7200 (WinXP) and ran your example - thankfully, it was simple and self-contained.



for /l %i in (1,1,1) do @start /b rscript -e "system.time(for(i in
1:1e4){fisher.test(matrix(c(200,300,400,500),nrow=2))})"

   user  system elapsed
  54.50    0.04   54.63

for /l %i in (1,1,2) do @start /b rscript -e "system.time(for(i in
1:1e4){fisher.test(matrix(c(200,300,400,500),nrow=2))})"

   user  system elapsed
  55.64    0.00   55.77
   user  system elapsed
  55.92    0.02   56.08

Test of multiple concurrent R processes on Windows XP on a dual-core m
+achine(R version 2.9.2)
[download]

So at least on Windows, two R processes *can* run alongside each other. I did not have the /affinity switch on XP so the apparently smaller 'concurrency overhead' may be explained by CPU thrashing.

The same thing on Linux, on my work E7300:

$ for i in $(seq 1 1); do Rscript -e 'system.time(for(i in 1:1e4){fish
+er.test(matrix(c(200,300,400,500),nrow=2))})' & done

   user  system elapsed 
 46.694   0.088  47.189 

$ for i in $(seq 1 2); do Rscript -e 'system.time(for(i in 1:1e4){fish
+er.test(matrix(c(200,300,400,500),nrow=2))})' & done

   user  system elapsed 
 48.007   0.060  48.188 
   user  system elapsed 
 47.838   0.072  49.487
[download]

One may conclude that an E7300 is measurably faster than an E7200 (big surprise there), that the 'concurrency overhead' is perhaps a bit larger on Linux, and most importantly, that the two concurrent R processes can use the two CPUs just fine.

BUT! (And picture me saying this just like Tim the Enchanter from MP&Holy Grail)

When I tried the specific R commands I need for my processing, I got this:


$ tail Rtest1.txt 

0.05
0.05
0.05
0.05
0.05
0.05
0.13
0.08
0.10
0.15

$ wc -l Rtest1.txt 

91135 Rtest1.txt

$ cp Rtest1.txt Rtest2.txt 

$ cat Rtest1.R

library(splines)
library(survival)
library(NADA)
outcome=scan("Rtest1.txt")
cenoutcome=rep(FALSE, length(outcome))
cenoutcome[outcome==min(outcome)]=TRUE
pyros=cenros(outcome,cenoutcome)
mean(pyros)
proc.time()

# Rtest2.R is the same, but it reads Rtest2.txt

$ Rscript Rtest1.R 2> /dev/null &

[1] 0.03195313
   user  system elapsed 
 22.201   4.096  26.293 


$ Rscript Rtest1.R 2> /dev/null & Rscript Rtest2.R 2> dev/null &

[1] 0.03195313
   user  system elapsed 
 40.358   6.328  46.839 
[1] 0.03195313
   user  system elapsed 
 39.706   6.044  48.128
[download]

So it is not R itself that has a problem with multiprocessing environments, nor is it my clumsy Perl threaded implementation: it is this specific R package.

BrowserUk, I sincerely thank you for taking interest in my silly little problem. You helped me a lot. I am indebted to you.