Useful fork with IPC on win32 ?

NetWallah has asked for the wisdom of the Perl Monks concerning the following question:

Esteemed Monks - I seek meditations and enlightenment - yet again - on a possibly futile attempt to use "fork" in win32.

This year-old discussion illustrates the issue I'm currently facing. That discussion, it seems to me remains unresolved. I'm running ActiveState perl 5.6.0 - attempting to use pipes limits me to 2 threads, after which the program hangs attempting to open the third pipe. I tried to use Net::Shared, as suggested in the discussion - This errors out with "Unknown error at fork-test.pl line 12" on the "store" (See code below).

I'm not hung up on using any particular mechanism - I'm just looking for a consistent, easy to use "for-dummies" type approach to doing useful work by forking multiple processes, and returning results to the parent process.

The model I'ld like is to have a work queue, and as each child-process gets idle, it grabs a unit of work (some kind of lock needed, of course). When the work queue is empty, everybody quits and goes home. The results of the work should be returned to the parent.

In truly glorified mode, I'd like to be able to keep starting child processes, until a specified resource limit is reached, say 75% memory and/or CPU - but that is dreaming - and while I'm at it, it would be nice if the child processes started self-terminating when resources exceed another threshold for a certain time.

Back in the real world -just some easy IPC with multiple child processes in Win32 would be great. Small, legible working examples would be appreciated.

use strict;
use Net::Shared;

my $num_children = 2;    # How many children we'll create
my @children;            # Store connections to them
$SIG{CHLD} = 'IGNORE';   # Don't worry about reaping zombies
#my $LF  =  0x0A ; # \012; ==  \cJ  ==  ASCII 10
my $listen = new Net::Shared::Handler;
my $new_shared = new Net::Shared::Local (name=>"new_shared");#, port=>
+3252, debug=>0);
$listen->add(\$new_shared);

$listen->store($new_shared, "");


for my $num (1..$num_children){
 print "Opening child $num...";

  defined(my $child = fork) or  die "fork() failed: $!";
  if ($child == 0){
     &childproc($num);
     exit;
  }else{
    print "Parent here .. In loop $num\n";
    push @children, {pid=>$child};
    print "parent added child $num ref \n";
   };
  }

&parent();
$listen->destroy_all;
exit 0; # End of prog
#############################

sub childproc(){
    my $param = shift();

    print STDERR "Child $param says Hello\n";
# Child simply echoes data it receives, until EOF
   my $var = "assigned at the child";
    $listen->store($new_shared, $var);

         exit;


}
#################
sub parent(){
  print "In parent process Child pids=@children \n";


  # Send some data to the kids
 for my $i (1..20){
     # pick a child at random
     my $num = int rand $num_children;
 }
   sleep 1;
   
  while ()
    {
        my $var = $listen->retrieve($new_shared);
        next unless $var;
        print "Parent says that \$var was \"", $var,"\".\n" if $var;
        last;
    }
       
  foreach(@children){

    my $endpid=waitpid $_->{pid}, 0 ;
    print "Parent process  endpid=$endpid Child($_->{pid}) status=$?"
        . "(div 256=" . $? / 256 .")\n";
  }
}
[download]

update (broquaint): added <readmore> tag

Comment on Useful fork with IPC on win32 ? Download Code

Replies are listed 'Best First'.
Re: Useful fork with IPC on win32 ? by Thelonius (Priest) on May 27, 2003 at 10:49 UTC
I have used fork() and INET sockets successfully on Windows, although I've never used more than 2 or 3 processes. I tried your program on ActivePerl 5.8 on Windows 2000 and it ran fine. I've been using Windows 2000 at work for a couple of years now and Windows XP at home. Back when I was running 95/98/ME, I had a lot more problems. You should consider upgrading to 5.8. You can also try running Perl under Cygwin (it's a free Unix-like environment for Windows based on the GNU tools). Cygwin runs great on Windows 2000 and XP. Once again, it's more problematic with 95/98/ME. It is much more compatible with Unix than Activeperl. You can have both Activeperl and Cygwin Perl installed at the same time--they go in separate directories. The fork emulation is much better--it's still slow compared to Unix, but fast enough for many uses. You could also use Win32::Process instead of fork() to start up brand new processes. You would pass a command-line argument telling the program that it is a child (and possibly giving a socket port to connect to). You can try debugging Net::Shared--just put a bunch of print statements in the modules and see exactly where it fails--you might be able to fix it, even without upgrading your Perl. The module files should be installed as: c:\Perl\site\lib\Net\Shared.pm c:\Perl\site\lib\Net\Shared\Handler.pm c:\Perl\site\lib\Net\Shared\Local.pm c:\Perl\site\lib\Net\Shared\Remote.pm Specifically, `sub store` is in Handler.pm and it calls a couple of subs in Local.pm. One other piece of advice is that you probably don't want to try to max out the number of processes that you are running. If you spend most of your time waiting for a response over the network, then maybe 10 to 15, but that may be pushing it. Your network connection bandwidth, CPU, and memory will probably limit you before that.	[reply] [d/l]
Re: Useful fork with IPC on win32 ? by meredith (Friar) on May 27, 2003 at 11:47 UTC
Are you stuck with 5.6.0? I'd download ActiveState 5.8.0 and use ithreads; there is a Thread::Queue object included in the core distro if you're using threads, you just have to create your shared queue(s) before breaking out. It is designed to be thread-safe, but you'll soon hear the howls of, "it's not production-ready" or "it's not stable yet". I haven't had too much trouble with threads myself though. I have a process that I was testing that would run about 60 days before strange stuff began happening (and that was probably my code ;) Anyway, you might check that. `mhoward - at - hattmoward.org`	[reply]
Re: Useful fork with IPC on win32 ? by zakzebrowski (Curate) on May 27, 2003 at 10:39 UTC
~~What you're doing sounds reasonable for unix, but not necessiarilly for win32.~~ If you don't get this to work... Check out the win32 section of cpan and specifically look at the win32::process module. On a sepereate note, depending on what you are doing with your children processes, consider using a database or irc for your ipc communication layer. I recently tried using the pircd server and the Net::IRC modules as a means of IPC, but I ran into issues when it took too long for me to process a message... Update: Added strike to text since people after me negated my response. ---- Zak Pluralitas non est ponenda sine neccesitate - mysql's philosphy	[reply]
Re: Useful fork with IPC on win32 ? by hangmanto (Monk) on May 27, 2003 at 16:36 UTC
You may also want to look at Win32::ProcFarm. You can install it via PPM from ActiveState's PPM Repository. This module handles the grunt work of spawning processes and handling the return values. Be sure to look at the examples provided. It's worked well for me.	[reply]
Re: Useful fork with IPC on win32 ? by Jenda (Abbot) on May 27, 2003 at 16:26 UTC
I have several multithreaded services written in Perl (originaly 5.6.1, now 5.8). I usualy use Win32::BiggerPipe (did not find time to make it CPAN ready) to send data to the children and Win32::Semaphore to control the access to the pipe(s). You may see an example (old, probably no longer functioning script to download images from eGroups/YahooGroups groups). It needs IThread::AtFork module (also not on CPAN). You can read up here on what is this module supposed to solve. I usualy only need to send commands from the main thread to the children, but you might keep the pipes open in the oposite direction as well and send data back. The only thing you have to keep in mind is that pipe reads are blocking under windows (select() doesn't work) so if you do not want the threads to block you have to use the Win32::Semaphores to signal it's safe to read from a pipe. You of course have to be carefull not to read too much or too little. Jenda Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. -- Rick Osborne Edit by castaway: Closed small tag in signature	[reply]