in reply to Splitting an array into subarrays of size ~n

If you just want working code, I have the perfect solution for you: Parallel::ForkManager. Perfect fit, so to speak

if you want to know more about fork and wait, just use "perldoc -f fork", "perldoc -f wait", "perldoc -f waitpid" and "perldoc perlipc". Or read the corresponding man pages of the underlying C library.

Basically you should be able to do up to 4 forks, then just wait until you wait()ed an equal number of times (or better until wait() returns -1). Getting the first 4 entries of an array is easy, just use my @part= splice(@array,0,4). How much you really get out of the array (because it may have only 1, 2 or 3 entries left) you will find out with scalar(@part)

Replies are listed 'Best First'.
Re^2: Splitting an array into subarrays of size ~n
by hotel (Beadle) on Mar 21, 2011 at 15:41 UTC

    Thanks for all of the answers.. I do not want to use Parallel::ForkManager because I am not quite sure how to ship it with my script (this is a homework, but do not worry, I am doing an "extra" feature in the homework so it is okay (I can run 20 children processes at the same time, no one cares, but i want to do ~4 by ~4 when possible).

    I am just not sure how to fork ~4 different processes. Below is the code I just wrote.. 
    while(scalar(@directories) > 0) {
    
    #Take first 4 when there are 4 or more elements in the array
    if(scalar(@directories)>=4){
        @part= @directories[0..3];
    }    
    
    #Take as many as possible when <4 elements in the array
    elsif(scalar(@directories)<4){
        @part = @directories [0..scalar(@directories)];

    }
    
    #Shorten the array by the number of elements taken in @part
    for(my $i=0; $i<scalar(@part); $i++) {
        shift(@directories);
    }
    
    #For each element in @part, create children processes
    #************************************
    foreach my $f (@part) {
        
        my $secdirtoget= "${passed_dir}/StretchingDecaAlanine/GMXCubic
+Box/Umbrella/${f}";
        chdir ($secdirtoget);
    
        system ("ln -s ../md_umbrella.mdp ../index.ndx ../topol.top ..
+/posre_CTerm.itp ../posre.itp .");
        system ("grompp -f md_umbrella.mdp -c pullconf.gro -n index.nd
+x");
        exec 'mdrun', '>logje 2>&1 &';
        
        chdir ("${passed_dir}/StretchingDecaAlanine/GMXCubicBox/Umbrel
+la");    
    }
    #************************************
}

    [download]
    The thing is that I want my script to wait until 4 or less mdrun calls in the foreach my $f (@part) finish doing their job, and then continue with the next 4 or less elements when a new partition in the beginning of the while is made.. Many thanks..
[reply]
[d/l]
[select]
      Hi again.. I guess I figured it out :-) The below seems to work: 
      sub mdrunner {
    
    my $passed_dir = $_[0];
    my $dirtoget="${passed_dir}/StretchingDecaAlanine/GMXCubicBox/Umbr
+ella";
     my (@directories, @part) = ();
     
     opendir(IMD, $dirtoget) || die("Cannot open directory");
 
     my @files= readdir(IMD);
     closedir(IMD);
     
     foreach my $g (@files) {
         push(@directories, $g) if ((-d $g) and ($g ne ".") and ($g ne
+ ".."));
     }
     
     while(scalar(@directories) > 0) {
        
        #Take first 4 when there are 4 or more elements in the array
        if(scalar(@directories)>=4){
            @part= @directories[0..3];
        }    
        
        #Take as many as possible when <4 elements in the array
        elsif(scalar(@directories)<4){
            @part = @directories [0..scalar(@directories)];

        }
         
         #Shorten the array by the number of elements taken in @part
         for(my $i=0; $i<scalar(@part); $i++) {
            shift(@directories);
        }
        
        #For each element in @part, create children processes
         foreach my $f (@part) {
             
             my $secdirtoget= "${passed_dir}/StretchingDecaAlanine/GMX
+CubicBox/Umbrella/${f}";
             chdir ($secdirtoget);
               
               system ("ln -s ../md_umbrella.mdp ../index.ndx ../topol
+.top ../posre_CTerm.itp ../posre.itp .");
               system ("grompp -f md_umbrella.mdp -c pullconf.gro -n i
+ndex.ndx");
               
               my $pid = fork();
               
               if ($pid == -1) {
                   die;
               } 
               
               elsif ($pid == 0) {
                   print "Running mdrun for within $f\n";
                   exec 'mdrun', '>logje 2>&1 &' or die;
              }
            
            chdir ("${passed_dir}/StretchingDecaAlanine/GMXCubicBox/Um
+brella");
         }
        while (wait() != -1) {}
        print "Done with mdrun in"."@part\n";
     }
}

      [download]
[reply]
[d/l]

        Good work. Two comments:

                if(scalar(@directories)>=4){
            @part= @directories[0..3];
        }    
        
        #Take as many as possible when <4 elements in the array
        elsif(scalar(@directories)<4){
            @part = @directories [0..scalar(@directories)];

        }

        [download]

        This is overkill, since you can reference non-existing array elements in perl without problems. So the following is equivalent:

               @part= @directories[0..3];

        [download]

        The line

        if ($pid == -1) {

        [download]

        should be

        if (not $pid) {

        [download]

        because fork returns undef on failure, unlike the C library fork which returns -1

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom