exec vs. system

Microcebus has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I have a perlscript called swap.pl an want to start it several times with another perl script to use my 4 CPU cores. Each time, the swap.pl script processes a different part of a given dataset. My code looks like:

foreach(1..$number_of_cpus)
    {
    open(TEMP_CTRL,">temp_ctrl");
    print TEMP_CTRL "$_"; # contains information for swap.pl
    close TEMP_CTRL;
    system("swap.pl"); # swap.pl reads temp_ctrl to know which part of
+ the dataset to process
    sleep(1);
    }
[download]

The problem is, that the system command waits until the process is finished, which means there is no parallelization at all. And the problem with exec command is that it does not return.

Can anyone help?

UPDATE: Thanks for the suggestions. I found

system(1,"swap.pl")
[download]

to work well.

Comment on exec vs. system Select or Download Code

Replies are listed 'Best First'.
Re: exec vs. system by derby (Abbot) on Jul 07, 2011 at 12:32 UTC
Well ... there are a number of ways to put swap.pl into a background process but you'll find the easiest approach to your problem is to use Parallel::ForkManager and totally avoid the use of an external file and/or an external program. -derby	[reply]
Re: exec vs. system by jethro (Monsignor) on Jul 07, 2011 at 12:29 UTC
system("swap.pl &") will put swap.pl into the background (a feature of the shell). By the way, wouldn't it be easier to give the data as parameter to swap.pl, i.e. system("swap.pl \"$_\" &") ? This would only work for non-binary data naturally and you would have to escape " \ and $ in the string	[reply]
Re^2: exec vs. system by jpl (Monk) on Jul 07, 2011 at 17:39 UTC
By the way, wouldn't it be easier to give the data as parameter to swap.pl, i.e. system("swap.pl \"$_\" &")? This would only work for non-binary data naturally and you would have to escape " \ and $ in the string. Not only easier, but more robust. Those of us who have been burned in almost every possible way cringe at the assumption that `swap.pl` will be done with the control file in less than one second, when it will be replaced by the control file for another instance. If the control data is too complex to pass as an argument, then give each control file a different name and pass the name to `swap.pl` as an argument.	[reply]
Re^2: exec vs. system by Microcebus (Beadle) on Jul 07, 2011 at 13:08 UTC
Thanks for that reply but system("swap.pl &") seems not to put the process into the background. The master script still waits untill swap.pl is finished.	[reply]
Re^3: exec vs. system by jethro (Monsignor) on Jul 07, 2011 at 14:30 UTC
Ah, so you probably are on Windows. At least I never heard of system(1,...) being sensible syntax, so probably this is an extension in your windows perl dialect	[reply]
Re^4: exec vs. system by jpl (Monk) on Jul 07, 2011 at 18:00 UTC
Re: exec vs. system by tokpela (Chaplain) on Jul 07, 2011 at 19:34 UTC
Based on the above responses (and the fact that you are using Windows), maybe using threads might be a better solution for you? See BrowserUK's posts for some good examples. Try this one which uses threads to run system commands or this one which uses Thread::Queue (but no system command).	[reply]
Re: exec vs. system by locked_user sundialsvc4 (Abbot) on Jul 07, 2011 at 12:34 UTC
The “ruling constraint” of your program is probably not CPU capacity: it is I/O. Throwing additional processes at an I/O-bound algorithm generally makes it dramatically slower, not faster.
Re^2: exec vs. system by Anonymous Monk on Jul 07, 2011 at 12:51 UTC
The “ruling constraint” of your program is probably not CPU capacity: On what information do you base that conclusion?	[reply]