in reply to Re: Re: Re: Re: IO::Select - is it right for this?
in thread IO::Select - is it right for this?

As promised, here is the code that works (well, sort of works anyway):
for ($i=0; $i<$g; $i++) { pipe ($rh, $wh) # both of these are generated in the loop if ($pid[$i]=fork()) { # parent process waitpid($pid[$i], 0); # wait for child close ($wh); while (<$rh>) { # gimme the output if ($_ =~ m/^Error/) { push (@error, $_); } elsif ($_ =~ m^Hits/) { push (@hits, $_); } else { push (@data, $_); } # each is a result from an engine } } else { close ($rh); open (STDOUT, ">&$wh"); open (OUT, "| $caller"); # call the handler with arguments close (OUT); exit(0); } }


This is part of a multi-headed search engine. The process is to go out to several search engines, poll them for results, and pass them back to the main program. The program is customized to allow from 10-50 results from each search engine. If I say get 10 or 20 from each engine, it works fine. But if I go for 30 or more, it chokes. Is it possible that I'm trying to pull too much through the pipe?

Yes, I know there is stuff missing, but nothing that is relevant to the question at hand. All of that I can't share here.

I should also point out that the idea is to get all the searches to run at the same time so hopefully the entire process will speed up.

Replies are listed 'Best First'.
Re: Re: IO::Select - is it right for this?
by Skeeve (Parson) on Oct 22, 2003 at 08:05 UTC
    I fear your version does not work in parallel!
    I just tested it with this variant:
    $g=10; $|=1; for ($i=0; $i<$g; $i++) { pipe ($rh, $wh); # both of these are generated in the loop if ($pid[$i]=fork()) { # parent process waitpid($pid[$i], 0); # wait for child close ($wh); while (<$rh>) { # gimme the output if ($_ =~ m/^Error/) { push (@error, $_); } elsif ($_ =~ m/^Hits/) { push (@hits, $_); } else { push (@data, $_); } # each is a result from an engine } } else { close ($rh); open (STDOUT, ">&$wh"); $|=1; print "I will wait for ",$i+2," seconds\n"; sleep $i+1; print "Good morning. I slept about ",$i+2," seconds\n"; exit(0); } } print @data;
    Two problems:
    1. It really doesn't work in parallel
    2. It doesn't give the output to the parent
    It produces this output:
    I will wait for 2 seconds Good morning. I slept about 2 seconds I will wait for 3 seconds Good morning. I slept about 3 seconds I will wait for 4 seconds Good morning. I slept about 4 seconds I will wait for 5 seconds Good morning. I slept about 5 seconds I will wait for 6 seconds Good morning. I slept about 6 seconds I will wait for 7 seconds Good morning. I slept about 7 seconds I will wait for 8 seconds Good morning. I slept about 8 seconds I will wait for 9 seconds Good morning. I slept about 9 seconds I will wait for 10 seconds Good morning. I slept about 10 seconds I will wait for 11 seconds Good morning. I slept about 11 seconds
    But the output should be something like this:
    I will wait for 2 seconds I will wait for 3 seconds I will wait for 4 seconds I will wait for 5 seconds I will wait for 6 seconds I will wait for 7 seconds I will wait for 8 seconds I will wait for 9 seconds I will wait for 10 seconds I will wait for 11 seconds Good morning. I slept about 2 seconds Good morning. I slept about 3 seconds Good morning. I slept about 4 seconds Good morning. I slept about 5 seconds Good morning. I slept about 6 seconds Good morning. I slept about 7 seconds Good morning. I slept about 8 seconds Good morning. I slept about 9 seconds Good morning. I slept about 10 seconds Good morning. I slept about 11 seconds
    which I produced with a variant of my former version:
    #!/usr/local/bin/perl use warnings; use strict; my $children= 10; my @out; my $first_child; $|=1; my $pid= open($first_child, '-|'); if ($pid) { # parent @out= <$first_child>; } else { my(@child, @childpid); for (my $i=0; $i<$children; ++$i) { $childpid[$i]= open($child[$i], '|-'); if ($childpid[$i]) { # do nothing yet } else { print "I will wait for ",$i+2," seconds\n"; sleep $i+1; print "Good morning. I slept about ",$i+2," seconds\n"; exit; } } foreach (@child) { close $_; } exit; } print @out;
      This is good, but not quite right. The array @out is working nicely.
      It seems that when I try to adapt this to my needs, I only get results from one of my search engines.

      I tried eliminating one of the exit; statements, but it doesn't quite work right. Eliminating the first one doesn't change anything. Eliminating the second does give me all the outputs, but give me a lot of additional garbage with it, and messes up the output. Here is the code as it stands now.

      my @out; my $first_child; $|=1; my $pid= open($first_child, '-|'); if ($pid) { # parent @out= <$first_child>; } else { my(@child, @childpid); for (my $i=0; $i<$children; ++$i) { $childpid[$i]= open($child[$i], '|-'); if ($childpid[$i]) { # do nothing yet } else { $call = <Something I make here>; # sorry, can't show that open (OUT, "| $call"); close (OUT); exit; } } foreach (@child) { close $_; } exit; }

      $call is a PERL call that is generated by using information gathered elsewhere in the program. It is built just prior to the call to run it, and will be different for each iteration of the for loop.
      Then I work with @out in the rest of the code. I'm thinking that the issue is with how I run $call, but I'm not sure how else to do this.

      IPC is new to my coding bag of tricks, so I appreciate all the input everyone has given.
        Doh!
        Forget that last post.
        I put the foreach(@child) loop in the wrong place. My bad. Sorry. :)
        This part:
        else { $call = <Something I make here>; # sorry, can't show that open (OUT, "| $call"); close (OUT); exit; }
        can be simplified to
        else { $call = <Something I make here>; # sorry, can't show that exec $call || die "Something went wrong!"; }
        But you should better consider using @call, putting each option and parameter of the call into it's own @call element and using the complete path for the first parameter, the program's name.
        For example instead of
        $call= "myperlscript -param1 $var1 -param2 $var2"; exec $call;
        use something like this:
        @call= ('/my/home/dir/myperlscript','-param1',$var1,'-param2',$var2); exec @call;