in reply to Sequential processing with fork.

Here's an example using Parallel::ForkManager that has come in very handy lately for these types of questions. It'll process $max_forks at a time. In the for loop, I've put in the number of clients (33). It'll process 10 at a time, calling do_something() for each one until all 33 are exhausted.

#!/usr/bin/perl

use warnings;
use strict;

use Parallel::ForkManager;

my $max_forks = 10;

my $fork = new Parallel::ForkManager($max_forks);

# on start callback

$fork->run_on_start(
    sub {
        my $pid = shift;
    }
);

# on finish callback

$fork->run_on_finish(
    sub {
        my ($pid, $exit, $ident, $signal, $core) = @_;
        if ($core){
            print "PID $pid core dumped.\n";
        }
    }
);

# forking code

for my $client (1..33){

    $fork->start and next;

    do_something($client);
    sleep(2);

    $fork->finish;

}

sub do_something {
    my $client = shift;
    print "$client\n";
}

$fork->wait_all_children;

[download]

-stevieb

UPDATE: I can't say for sure, but after removing the sleep statement, the output hints at the fact that it'll add more into the queue before previous ones are finished as long as the max count doesn't go over 10. I'm not 100% sure of this though.

UPDATE 2: According to the Parallel::ForkManager docs, it does indeed throw another proc onto the heap after each one finishes. This number is configurable:

wait_for_available_procs( $n )

    Wait until $n available process slots are available. If $n is not 
+given, defaults to 1.

[download]

Replies are listed 'Best First'.
Re^2: Sequential processing with fork.
by ikegami (Patriarch) on Aug 04, 2015 at 20:03 UTC
    That's the most useless run_on_finish possible. Worse than none at all. 
    #!/usr/bin/perl

use warnings;
use strict;

use Parallel::ForkManager qw( );

use constant MAX_WORKERS => 10;

sub work {
    my ($client) = @_;
    print("$client start...\n");
    sleep(3 + int(rand(2)));
    print("$client done.\n");
}

{
   my $pm = new Parallel::ForkManager(MAX_WORKERS);

   # Optional.
   $pm->run_on_finish(sub {
      my ($pid, $exit, $ident, $signal, $core) = @_;
      if ($signal) {
         print("Client $ident killed by signal $signal.\n");
      }
      elsif ($exit) {
         print("Client $ident exited with error $error.\n");
      }
      else {
         print("Client $ident completed successfully.\n");
      }
   });

   for my $client (1..33){
      $pm->start($client) and next;
      work($client);
      $pm->finish();
   }

   $pm->wait_all_children();
}

    [download]
[reply]
[d/l]
[select]

      Yes, I realized I had emptied it out previously after the fact. I'll leave it as is so this post retains context. Thanks for pointing it out.

[reply]
Re^2: Sequential processing with fork.
by Kelicula (Novice) on Aug 04, 2015 at 20:38 UTC

    Yes, that's exactly what I want...as soon as ONE process ends, start another one!!

    I just can't ever have more than ten running simultaneously...

    ~~~~~~~~~ I'm unique, just like everybody else! ~~~~~~~~~
[reply]
Re^2: Sequential processing with fork.
by Kelicula (Novice) on Aug 08, 2015 at 18:49 UTC

    Stevieb,

    Thank you! Your code seemed to point me in the right direction and do just what I needed, (after trying several other examples) the only problem is for what ever reason it doesn't progress to the next process after the initial group is started.

    As I mentioned earlier I'm actually calling on another file and passing it the client ID via @ARGV, so it's not like it's simply a lexical scope that's being forked.

    I've tried using exec, system to no avail. After one of the original group processes ends, it doesn't start another one to replace it or create a new one to run the next client.

    After all the initial processes finish the main program just ends. :-( Any idea what could be going on?

    Here's what I'm working with..

    #!/usr/bin/perl

use warnings;
use strict;
use Uber qw / client_list /;

use Parallel::ForkManager;

my $max_forks = 2; # Changed to 2 just to see if it would go to the ne
+xt one...
my $clients   = client_list();

my $fork = new Parallel::ForkManager($max_forks);


for my $client ( @$clients ){

    $fork->start( $client->{id} ) and next;

    do_something( $client->{id} );

    $fork->finish;
    
}

sub do_something {
    my $client = shift;
    system( "perl", "C:/Path/To/Folder/process.pl", "$client" );
}

$fork->wait_all_children

    [download]
[reply]
[d/l]
Re^2: Sequential processing with fork.
by Kelicula (Novice) on Aug 04, 2015 at 19:56 UTC
    Thank you for the example, but yes, my system can only handle around ten running at a time.
    ~~~~~~~~~ I'm unique, just like everybody else! ~~~~~~~~~
[reply]

      Then my code does exactly what you need. You can modify or even remove the $fork->run_on_* subs if you don't need the PIDs or don't care if any of the procs failed.

[reply]
[d/l]

In Section Seekers of Perl Wisdom