I think that I've pretty much confirmed that R does some serialisation of resource usage across concurrent process instances. Even though the evidence is not as clear cut as I would like. In the following console trace, I start 1, 2, 3 & 4 concurrent copies of R performing a ~60 second calculation and timing it with its own built-in timer. I set the affinity of each process to 1 of my cpus to prevent cpu thrash:
for /l %i in (1,1,1) do @start /b /affinity %i rscript -e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n +row=2))})" user system elapsed 62.24 0.00 62.68 for /l %i in (1,1,2) do @start /b /affinity %i rscript -e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n +row=2))})" user system elapsed 65.49 0.00 65.60 user system elapsed 65.19 0.01 66.13 for /l %i in (1,1,3) do @start /b /affinity %i rscript -e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n +row=2))})" user system elapsed 65.61 0.06 65.94 user system elapsed 65.75 0.03 98.98 user system elapsed 65.55 0.00 99.30 for /l %i in (1,1,4) do @start /b /affinity %i rscript -e"system.time(for(i in 1:1e4){fisher.test(matrix(c(200,300,400,500),n +row=2))})" user system elapsed 68.83 0.00 69.81 user system elapsed 70.59 0.00 72.71 user system elapsed 67.30 0.03 101.99 user system elapsed 67.22 0.00 102.65
The cpu times of the 3 processes all show the ~5% concurrency overhead--probably indicative of some internal polling for resource--but the elapsed times show much greater overhead--nearly 70%.
That pretty much nails it (for me) that there is some one-at-a-time resource internal to R that concurrent processes compete for. And the more there are competing, the greater the cost of that competition.
All of which probably reflects Rs long history and its gestation in the days before multi-tasking concurrent cpu-bound processes was a realistic option.
Note: It could be that the shared resource is something simple like the configuration file or history file or similar; and that with the right command line options to disable the use of those, the overhead can be avoided. I haven't explored this. It might be better to ask the guys that know rather than testing random thoughts.
Whilst looking around I did come across Rmpi which might be a better approach to your problem. Though it appears to be aimed at spreading the load of single calculations over multiple cpus or boxes, rather than running multiple concurrent calculations. You'd need to read a lot more about than I've bothered with :)
In reply to Re^9: Using threads to run multiple external processes at the same time
by BrowserUk
in thread Using threads to run multiple external processes at the same time
by Anonymous Monk
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |