Re^3: Parallelization of heterogenous (runs itself Fortran executables) code

Here is a version using Text::ParseWords and a modified spawn subroutine, which uses exec in scalar context. This allows parameterized command execution.

#!/usr/bin/perl

use Getopt::Std;
use Text::ParseWords;
my %opt = (n => 1);
getopts('r:n:v', \%opt) or usage();

my $cmd = shift;
@ARGV = shuffle(@ARGV) if $opt{r};

my %pid;
while (@ARGV) {
  if (keys(%pid) < $opt{n}) {
    $pid{spawn($cmd, map { shellwords($_) } shift @ARGV)} = 1;
  } else {
    delete $pid{wait()};
  }
}

1 while wait() >= 0;

sub spawn {
  my $pid = fork;
  die "fork: $!" unless defined $pid;
  return $pid if $pid;
  warn "@_\n" if $opt{v};
  exec qq(@_);
  die "exec: $!";
}

sub usage {
  print STDERR "Usage: $0 [-n N] [-r] [-v] command arg1 arg2...
Run command arg1, command arg2, etc., concurrently.
Run no more than N processes simultaneously (default 1)
    -r: run commands in random order instead of specified order (unimp
+l.)
    -v: verbose mode
";
  exit 1;
}
[download]

The following snippet will ping google.com and amazon.com once in the first chunk and yahoo.com in the second.

runN -v -n 2 'ping -n 1' google.com amazon.com yahoo.com
ping -n 1 google.com
ping -n 1 amazon.com

Ping google.com [72.14.207.99] mit 32 Bytes Daten:

Antwort von 72.14.207.99: Bytes=32 Zeit=116ms TTL=242

Ping-Statistik für 72.14.207.99:
    Pakete: Gesendet = 1, Empfangen = 1, Verloren = 0 (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 116ms, Maximum = 116ms, Mittelwert = 116ms

Ping amazon.com [72.21.210.11] mit 32 Bytes Daten:

ping -n 1 yahoo.com

Ping yahoo.com [66.94.234.13] mit 32 Bytes Daten:

Antwort von 66.94.234.13: Bytes=32 Zeit=185ms TTL=54

Ping-Statistik für 66.94.234.13:
    Pakete: Gesendet = 1, Empfangen = 1, Verloren = 0 (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 185ms, Maximum = 185ms, Mittelwert = 185ms
Zeitüberschreitung der Anforderung.

Ping-Statistik für 72.21.210.11:
    Pakete: Gesendet = 1, Empfangen = 0, Verloren = 1 (100% Verlust)
[download]

print+qq(\L@{[ref\&@]}@{['@'x7^'!#2/"!4']});

Comment on Re^3: Parallelization of heterogenous (runs itself Fortran executables) code Select or Download Code

Replies are listed 'Best First'.
Re^4: Parallelization of heterogenous (runs itself Fortran executables) code by Dominus (Parson) on Nov 21, 2007 at 22:13 UTC
Here is a version using Text::ParseWords and a modified spawn subroutine, which uses exec in scalar context. I think that was an extremely bad move, because now you have to worry about shell metacharacters in arguments, which you formerly did not. For example, consider: `runN echo ''` [download] We are expecting this to run one `echo` to print out a star, but with your implementation, it does not; the quotes are stripped off and it echoes a list of the filenames in the current directory. Similarly, suppose a file in the current directory is named `foo bar`. Then `runN command ` will run the command not with the argument `foo bar` but with the two arguments `foo` and `bar`. Now let's suppose there is a file in the current directory named `rm -rf /`. (Note backquotes!) Then running your version of `runN command *` will erase the entire filesystem. I wrote a blog entry about the perils of multiple shell evaluation in this context. I said: My fear was that by introducing a double set of shell-like interpretation, I'd be opening a horrible can of escape character worms and weird errors, and my hope was that if I ignored the issue the problems might be simpler, and might never arise in practice. This is exactly the sort of thing I was worried about. Your implementation certainly proves that I was correct about at least the first part of this prediction! -- Mark Dominus Perl Paraphernalia	[reply] [d/l]

Replies are listed 'Best First'.

Re^4: Parallelization of heterogenous (runs itself Fortran executables) code
by Dominus (Parson) on Nov 21, 2007 at 22:13 UTC

Here is a version using Text::ParseWords and a modified spawn subroutine, which uses exec in scalar context.

runN echo '*'
[download]

echo

Similarly, suppose a file in the current directory is named foo bar. Then runN command * will run the command not with the argument foo bar but with the two arguments foo and bar.

Now let's suppose there is a file in the current directory named `rm -rf /`. (Note backquotes!) Then running your version of runN command * will erase the entire filesystem.

I wrote a blog entry about the perils of multiple shell evaluation in this context. I said:

My fear was that by introducing a double set of shell-like interpretation, I'd be opening a horrible can of escape character worms and weird errors, and my hope was that if I ignored the issue the problems might be simpler, and might never arise in practice.

--
Mark Dominus
Perl Paraphernalia

[reply]
[d/l]