Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

i'm building a networked app, and one request to a server takes 2-10 seconds, but since i need up to 100 replies from the server, i thought i'd use 100 children to get 100 replies all at the same time, but something isn't right and it gets pretty carried away sometimes. for the most part, this seems alright ran through a console, but accessing it via a browser attempts to kill the system.

i'd like the main sub to fork out as many children as needed, one for each search param, and collect all of the results and return them. each child really should return a value within 15 seconds or return bad. any help at all would be greatly appreciated, i've always been terrible at ipc.

use strict; use warnings; use Data::Dumper; use CGI::Carp qw/fatalsToBrowser/; use CGI; my $cgi = new CGI; unless ($cgi->param()) { print "Content-type: text/html\n\n"; print "<form action=$ENV{'SCRIPT_NAME'}><input type=text name=test> +<input type=submit></form>"; } else { my $param = $cgi->param('test'); my @patterns = find_patterns($param); print Dumper( lookup(@patterns) ); } #---------------------------------- sub lookup(@) { BEGIN { unshift @INC, "/home/test/lib" } use XML_Client qw(:default); use Data::Dumper; $SIG{CHLD} = \&sig_chld; my @search = @_; my %return; my @spawn; my $XML_Client = new XML_Client(); $XML_Client->login; use IO::Handle; pipe(READER, WRITER); #WRITER->autoflush(1); for (0..$#search) { my $pid = &child($search[$_], $XML_Client); push @spawn, $pid; } close WRITER; local $SIG{ALRM} = sub { die "timeout\n" }; alarm 20; waitpid($spawn[-1],0); for (0..$#search-1) { my $feedback = <READER>; chomp $feedback; $return{$_} = $feedback; } alarm 0; return %return; END { $XML_Client->logout if $XML_Client } #------------------------------- sub child($$) { my $search = shift; my $XML_Client = shift; my $return; unless (my $pid = fork) { close READER; local $SIG{ALRM} = sub { die "child timeout\n" }; my $lookup_data = { action => "lookup", attributes => { search => $search, } }; my $lookup_results; eval { local $SIG{ALRM} = sub { die "child timeout\n" }; alarm 15; $lookup_results = $XML_Client->send_cmd( $lookup_data ) or + warn PROG $!; alarm 0; }; if ($@) { #timed out die unless $@ eq "child timeout\n"; } else { print WRITER $lookup_results->{status} . "\n"; exit 1; } return $pid; } #----------------------------------- sub sig_chld { use POSIX ":sys_wait_h"; my $child = waitpid(-1, WNOHANG); $SIG{CHLD} = \&sig_chld; }

Replies are listed 'Best First'.
Re: IPC - child reporting
by Zaxo (Archbishop) on Jun 04, 2004 at 12:26 UTC

    Take a look at Parallel::ForkManager. You can set a maximum number of concurrent children and loop as many lookups as you like. P::FM takes care of all the messy parts.

    After Compline,
    Zaxo

Re: IPC - child reporting
by exussum0 (Vicar) on Jun 04, 2004 at 10:24 UTC
    Define carried away? You say you access it via a browser and things aren't ok. What error is in your web server's error log? (I'm assuming you are getting an HTTP 500).

    For a test, could you scale it back to just 2 and push forward? It also could be that you are forking one child for each query. Forking is expensive, so maybe you can fork a max of N processes and just eat the delay time in between for getting responses? 200 querries/20 processes would leave you 10 qpp, and at 10 seconds max for each one, 2 minues for something you do once in a while isn't bad.

    Also, what is this for? Maybe you can optimize other ways.

    Bart: God, Schmod. I want my monkey-man.

      sorry. i'm not getting a 500, it does return ok. if i pass 5, it dumps var0-4, which is fine. if i pass more than 5, it appears to call the script again for each multiple of 5:
      $VAR1 = '6'; $VAR2 = undef; $VAR3 = '11'; $VAR4 = undef; $VAR5 = '3'; +$VAR6 = undef; $VAR7 = '7'; $VAR8 = undef; $VAR9 = '9'; $VAR10 = unde +f; $VAR11 = '2'; $VAR12 = undef; $VAR13 = '12'; $VAR14 = undef; $VAR1 +5 = '8'; $VAR16 = undef; $VAR17 = '4'; $VAR18 = undef; $VAR19 = '1'; +$VAR20 = undef; $VAR21 = '0'; $VAR22 = undef; $VAR23 = '10'; $VAR24 = + undef; $VAR25 = '13'; $VAR26 = undef; $VAR27 = '5'; $VAR28 = undef; +Content-type: text/html $VAR1 = '6'; $VAR2 = undef; $VAR3 = '11'; $VA +R4 = undef; $VAR5 = '3'; $VAR6 = undef; $VAR7 = '7'; $VAR8 = undef; $ +VAR9 = '9'; $VAR10 = undef; $VAR11 = '2'; $VAR12 = undef; $VAR13 = '1 +2'; $VAR14 = undef; $VAR15 = '8'; $VAR16 = undef; $VAR17 = '4'; $VAR1 +8 = undef; $VAR19 = '1'; $VAR20 = undef; $VAR21 = '0'; $VAR22 = undef +; $VAR23 = '10'; $VAR24 = undef; $VAR25 = '13'; $VAR26 = undef; $VAR2 +7 = '5'; $VAR28 = undef; Content-type: text/html <etc, there's a lot more> Software error: timeout
      i can't see a possible reason for the extra "Content-type". the actual error appears to be either the main process or the children not dying when their time is up.