avi_learns_perl has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks,

I am pretty confused about perl forks...I am pretty new to perl so please be gentle :) ..... i shall define my problem. I need to call three different systems in a cluster all of which together need to execute n number of iterations of a single process individually and have different input data. They need to run simultaneously and without repetitions. ex: 

for(i=0;i<n;;)
{
execute command in system 1 input data is array[i];
execute command in system 2 input data is array[i+1];
execute command in system 3 input data is array[i+2];
i+3;
}

[download]
P.S. Here if system 2 finishes first then the next data in array should be sent to system 2. and no system should EVER be idle till no input data is remaining.

PLEASE SMS (Save My Soul)

Phew....If any queries or this seems disastrously complicated please let me know..I have been stuck at this point for over two days...and am looking for the monks to shed light for this disheartened soul..:(

Replies are listed 'Best First'.
Re: Forking problem UPDATED
by Moron (Curate) on Mar 06, 2007 at 13:07 UTC
    Forks spawn processes on the same system whereas Cluster::Init can spawn the processes elsewhere in a cluster - maybe that will get you started.

    -M

    Free your mind

[reply]
Re: Forking problem (manager)
by tye (Sage) on Mar 06, 2007 at 16:32 UTC

    I was going to propose the parent populate the array, create a pipe, create N children, and do print PIPE "$_\n" for 0..$#array; so each child could read (atomically) the index to the next data to be processed.

    But it looks like Parallel::ForkManager already pretty much does that for you so perhaps you should just use that.

    - tye        

[reply]
[d/l]
      i thought abt Parallel::ForkManager but my problem is that i need the pipes to be recreated dynamically such that after each child finishes there should be another one created...
[reply]

        after each child finishes there should be another one created

        That's exactly what Parallel::ForkManager does.

        Say you have 5 tasks, and you set a max of 3 children. ForkManager will create 3 children and assign them tasks 0, 1 and 2. When one of these end, another child is created for task 3. When one of child ends, another child is created for task 4. Having no more tasks to assign, it waits for the 3 children to end.

        The implementation is very simple: 

        use Parallel::ForkManager qw( )

use constant MAX_PROCESSES => 3;

my $pm = Parallel::ForkManager->new(MAX_PROCESSES);
foreach $data (@array) {
   # Forks and returns the pid for the child:
   my $pid = $pm->start and next; 

   ... do some work with $data in the child process ...

   $pm->finish; # Terminates the child process
}

        [download]
[reply]
[d/l]
Re: Forking problem UPDATED
by kyle (Abbot) on Mar 06, 2007 at 13:08 UTC

    What do you have so far?

    It sounds as though you need to fork twice. The parent and two children would each serve the inputs to one of your systems.

[reply]
Re: Forking problem UPDATED
by avi_learns_perl (Novice) on Mar 06, 2007 at 13:11 UTC
    @ above responders.... This thing needs to work in a grid environment....a cluster of clusters...so basically i am just doing job submission to the grid...my problem comes in making sure all three run paralelly and one doesnt wait for the other to complete
[reply]
      How are you submitting these jobs?

      -M

      Free your mind

[reply]
        Is there a way i can set variables in the forked process which will be updated into the parent process once the fork job is completed?
[reply]
        im using the system command to do that....
[reply]

Back to Seekers of Perl Wisdom