Consider the following bit of code: 
for ($i=0; $i<$g; $i++) {
  pipe ($rh, $wh)          # both of these are generated in the loop
  if ($pid[$i]=fork()) {   # parent process
    waitpid($pid[$i], 0);  # wait for child
    close ($wh);
    while (<$rh>) {        # gimme the output
      if ($_ =~ m/^Error/) { push (@error, $_); }
      elsif ($_ =~ m^Hits/) { push (@hits, $_); }
      else { push (@data, $_); }  # each is a result from an engine
    }
  }
  else {
    close ($rh);
    open (STDOUT, ">&$wh");
    open (OUT, "| $caller");  # call the handler with arguments
    close (OUT);
    exit(0);
  }
}

[download]


This is part of a multi-headed search engine. The process is to go out to several search engines, poll them for results, and pass them back to the main program. The program is customized to allow from 10-50 results from each search engine. If I say get 10 or 20 from each engine, it works fine. But if I go for 30 or more, it chokes. Is it possible that I'm trying to pull too much through the pipe?

Yes, I know there is stuff missing, but nothing that is relevant to the question at hand. All of that I can't share here.

I should also point out that the idea is to get all the searches to run at the same time so hopefully the entire process will speed up.

In reply to Concurrent Processes by mcogan1966

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.