I think you likely want to look into one of the available jobqueues for that. The simple one would be Parallel::ForkManager for handling jobs within one process. There are Gearman and TheSchwartz for machine-spanning job queueing.

Hi Corion,

Well i can use the Parallel::ForkManager for spawning children agreed though i m able to do that thru my way as well .. but the real problem is after i have created threads into each child process .Lets say that the no of threads are 3, now each thread is processing data that it has been passed, in batches after finishing one batch it lets the main thread know that it has done its first batch and suspends its operation.Now the main thread would launch a new thread to work on the result of the 1st thread and resume the suspended thread to continue its work.

The app that i m designing has to launch multiple queries on some server.The queries have placeholders(e.g <CLIENT> is a place holder) which are replaced at runtime. There are numerous queries and clients.A serial/single process approach would not help in speeding up things, so i thought of this model.Script starts forks 5 chidren each child picks a query and client list and then constructs 3 threads and passes divided client list to each thread.Now inside each thread the thread reads the query replaces the client fires to prod and test. AFter this it lets the main thread know that it is done and its time the main thread does comparison.The main thread constructs a new thread to do that and lets its sub thread continue with the batch.

The rationale behind launching multiple threads doing similar and different task is to increase the speed by parallel processing and reducing dependecy . As in the above case when the results as availble y wait in the same thread for comparison instead let the main thread know, which would launch a new comarison thread and let the old thread continue firing query

Thanks

I don't know much, if anything, about forking processes or creating threads in perl.

However, if each child process, i, is creating Y threads, then should the threads perhaps be created in the child process?

Then you'd have X(i)Y(j) for the thread id - X(i) being the id of the child and j being the id of the j'th thread created by child i, i=1..X and j=1..Y.

Think of it as a matrix or table with i,j identifying the matrix element or table cell running.

It's not clear to me if you need just all threads of child X(i) to be able to communicate, but they don't need to communicate with the threads of other children X(k),k=0..X, k!=i, i.e.

X(i)Y(j) can communicate with X(i)Y(k), k!=j but not X(k)Y(*), k!=i.

Or if you need all threads to be able to communicate to all other threads created by other children, i.e.

X(i)Y(j) can communicate with X(u)Y(v) and i,u=1..X, j,v=1..Y although i!=u and j!=v might still apply.

Hi LesleyB,

By spawning a child process using fork i m duplicating the perl interpreter which is running in a separate process. So if i have spawned 3 children total no of perl interpreter running would be 4 one for parent + 3 for the children.

when i m creating thread inside the child process forked above, no separate process is made. They are subprocesses of the child process.Now i want a way to communicate between these subprocesses and the process that has launched them.

Thanks

Don't have a chance to look at your source at this second, but depending on what your trying to use IPC for, these may be useful:

1. take a look at zentara's excellent thread primer -- he uses shared items to communicate items within threads. http://perlmonks.org/?node_id=691785

2. take a look at the source in http://perlmonks.org/?node_id=704138 where i use the Threads::Queue to do IPC.

It would be much easier to help you if you would post the code you have.

use strict;
use threads;
use threads::shared;
use Thread::Semaphore;
use POSIX;

my @clientList = qw(client1 client2 client3 client4 client5 client6 cl
+ient7 client8 client9 client10 client11 client12); my @dmServers = qw
+(A B C); my @queryPath = qw(AA BB CC DD EE FF); my @shared; my $query
+Index = 0; my $maxnoofThreads = 2; my %sharedHash : shared;

for (my $i = 0;$i < @dmServers;$i++)
{
        my $cpid = fork();
        die unless defined $cpid;
        if (! $cpid)
            {
                    # This is the child
                    # my $wait = int rand 10;
                    # sleep $wait;
                     callChild($queryPath[$queryIndex++],$dmServers[$i
+],$i);
                    #print "Child $$ exiting after seconds\n";
                    exit;
            }

}

# Just parent code, after this
while ((my $cpid = wait()) != -1)
     {
             #print "Waited for child $cpid\n";
         #print " length of array shared",length(@shared);
     }
#print "Parent Exiting\n";
sub callChild
{
    my $query = shift;
    my $dmserver = shift;
        my $i = shift;
        #print " query fired $query on dmserver $dmserver index $i\n";
        my @clientList_temp = @clientList;
        for(my $index = $i; $index < @queryPath;$index = $index + @dmS
+ervers)
        {
                  print "process $$ query at $index is ",$queryPath[$i
+ndex],"\n";
            # $shared[$i] = $i;
            # sleep rand(10);
              for(my $a;$a <= $maxnoofThreads;$a++)
                {
                    my $sema = Thread::Semaphore->new();
                    my @temp = splice(@clientList_temp,0,ceil(@clientL
+ist/$maxnoofThreads));
                    my $thr = thread->new(\&worker,$sema,\@temp,$query
+Path[$index]);
                    #if( defined %sharedHash)
                    #  {
                    #    my $thrnew = thread->new(\&compare);
                    #    $sema->up;
                    #  }
                }
                foreach my $thr (threads->list)
                       {
                             # Don't join the main thread or ourselves
                             if ($thr->tid && !threads::equal($thr, th
+reads->self))
                                {
##########Not Sure #######                        foreach my $client (
+keys %sharedHash)
                                  {
                                     print " client $client\n";
                                     my $thrnew = thread->new(\&compar
+e,$client);
                                       $sema->up;
                                  }
                                        $thr->join;
                                }
                           }

        }
    #$shared[$i] = undef;
}

sub worker
{
    my $sema = shift;
    my $myClientArrayRef = shift;
    my $query = shift;
      for(my $i;$i < @$myClientArrayRef;$i++)
        {
            print " $$myClientArrayRef[$i] client \n";
            sleep 5;
            my $thrid = threads->tid;
            $sharedHash{$$myClientArrayRef[$i]} = $thrid;
            shift @$myClientArrayRef;
                $sema->down;
        }
}

sub compare
{
    my $client = shift;
    print " hash value ",$sharedHash{$client},"\n";
}
[download]

Okay, I guess the question I have is, why do you need to fork before creating threads? Are you running this on win32? some nix flavor? If its win32, then the threads and actually forks, and using threads aren't really saving you much of anything

Note, in the code you've posted, you have a scope problem:

            if ( $thr->tid && !threads::equal( $thr, th +reads->self )
+ ) {
##########Not Sure #######
                foreach my $client ( keys %sharedHash ) {
                    print " client $client\n";
                    my $thrnew = thread->new( \&compare, $client );
                    $sema->up;
                }
[download]

$sema is not defined in this context, its local to the previous loop. there are numerous other bugs, but i'll leave that to you to find ;)

Again, I would refer you the the thread primer http://perlmonks.org/?node_id=691785 and the like http://perlmonks.org/?node_id=704138, but I've made a stab at doing what I _think_ you are looking to have done

Read more... (4 kB)
Note that the control structure of what I've done is _extremely_ sloppy and I would not reccommend it.

note that i don't quite understand why you need to thread your compares, but, I guess if that is also a time consuming operation, it could make sense

also note that I made your worker sub and compare sub local to the sub that is running in the fork. I did this for my own readability, as well as to remind myself that they are really running in the scope of the forked process and not the main program

In that vein, in my shared hash, I did not need the pid as a key in the hash, I just put it there for clarity, each forked process will be local unto itself, only copying the values, etc that are available at the time the process is forked (at least the way I understand the world

anyhow, hope this is helpful

Take a look at mutexes in perl e.g. APR::ThreadMutex or Threads::Shared

If you need to share data between threads then the principle is to mutually exclude all other threads from modifying that data while you transfer it.

Just set up a mutual exclusion and be religious about gaining and releasing it. You may not need to gain and release in the same subroutine.