Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

•Re: Run N similar tasks in parallel

by merlyn (Sage)
on Feb 20, 2003 at 14:24 UTC ( [id://237089]=note: print w/replies, xml ) Need Help??


in reply to Run N similar tasks in parallel

Here's the pattern I used for that on a recent project -- it's a bit more robust than yours.
use POSIX ":sys_wait_h"; my @tasks = (1..398); my %kids; { while (@tasks and keys %kids < 5) { $kids{fork_a_task(shift @tasks)} = "active"; } { my $pid = waitpid(-1, 0); if ($pid == -1) { %kids = (); } else { delete $kids{$pid}; } } redo if @tasks or %kids; } sub fork_a_task { my $i = shift; my $pid = fork; return $pid if $pid; unless (defined $pid) { warn "cannot fork: $!"; return 0; } ## do stuff for task $i goes here... exit 0; }

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

Replies are listed 'Best First'.
Re: &bull;Re: Run N similar tasks in parallel
by bronto (Priest) on Feb 20, 2003 at 14:47 UTC

    ++merlyn for checking fork's return value, that I forgot.

    Please, can you explain why your approach is more robust? I am really interested in it!

    Ciao and thanks!
    --bronto


    The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
    --John M. Dlugosz
      By checking the possible failure of fork, I never "think" I'm working on five kids when only two have really started.

      By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

      -- Randal L. Schwartz, Perl hacker
      Be sure to read my standard disclaimer if this is a reply.

        By checking the value of waitpid for a possible -1 return, I never "think" I have kids left when I really don't. The -1 force-resets the hash.

        Good, I should check for the return value of wait as well. Thanks.

        Sorry if I am bothering you about this, but I am missing an information: why do you prefer waitpid to wait? Flexybility? Possibility to make waitpid a non-blocking call? What?

        Thanks in advance.

        --bronto


        The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
        --John M. Dlugosz

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://237089]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (4)
As of 2024-04-19 03:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found