EchoAngel has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I'm still new to perl and it's language. Let say
foreach (0 .. 100) { system(" do something here ... long time "); }
what stuff can I do to make it execute in parallel? When I run my thing like this, it's like a linear path and takes a LONG TIME. What is the syntax/examples?

Replies are listed 'Best First'.
Re: Fork/Child Question
by Zaxo (Archbishop) on Jul 27, 2004 at 20:29 UTC

    The fork function is behind anything that can do that aside from threads. That is because fork creates a new process with the same environment the original process has. It therefore returns twice, returning 0 in the child process, and the child's pid or undef in the parent. The undef return indicates failure of fork.

    The basic receipe is to branch the execution stream after fork, with the parent going on and the child directed to do it's job and exit. The exec function is useful if the child is to do a system call.

    To avoid zombie processes, either set up an appropriate SIGCHLD handler or call wait/waitpid after the kids are launched.

    There is a limit to the number of child processes you can have, or should want, so big jobs may require the kind of throttling Parallel::ForkManager gives you.

    The receipe:

    $SIG{CHLD} = 'IGNORE'; # check if it works on your system for (1..100) { my $pid = fork; next if $pid; # in parent, go on warn($!), next if not defined $pid; # parent, fork errored out exec @cmd; # in child, # go do @cmd and don't come back }

    After Compline,
    Zaxo

Re: Fork/Child Question
by Paladin (Vicar) on Jul 27, 2004 at 20:03 UTC
Re: Fork/Child Question
by pbeckingham (Parson) on Jul 27, 2004 at 21:14 UTC

    To make this execute in parallel, you can fork, or you can thread. While threads are more lightweight than forking, they are not without their own overhead.

    It is tempting sometimes to want to parallelize certain operations, but typically there are limited CPUs in the machine, limited disk controllers etc, that give you decidedly non-linear characteristics. Good candidates for parallelizing are computationally distinct processes, asynchronous processes and the like. Operations that use the same piece of memory, or the same disk head are less likely to benefit. Don't forget that threading and parallelization are largely (but not completely) an illusion the OS creates for you.

    Nothing to do with your question per se, but the range

    0 .. 100
    contains 101 elements, not 100 - a trap I keep falling into, and is something to look out for.

Re: Fork/Child Question
by beable (Friar) on Jul 28, 2004 at 01:39 UTC

    Here's how you can do it using threads, assuming that your Perl has threading built in. (run perl -V to check if you can use threads (look for usethreads=define)).

    #!/usr/bin/perl use strict; use warnings; use threads; foreach (0..100) { # create a thread to call the "run_system" subroutine # to run "ls -la" on the OS my $thread = threads->create("run_system", "ls -la"); $thread->detach; } sub run_system { my $cmd = shift; system($cmd); } __END__

      Or as a one-liner :)

      perl -Mthreads=async -le"async{ system 'dir'; } for 1 .. 100"

      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
      "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon
Re: Fork/Child Question
by gaal (Parson) on Jul 28, 2004 at 04:30 UTC
    In this particular case, if you're on unix, you can simply start the job in the background:

    foreach (0 .. 100) { system(" do something here ... long time & "); }
    The only thing I changed from your code was to add an amersand at the end of the command to be executed by system. This should also work on cygwin, but generally not on other Windows perls.