How does a web server "know" when a CGI script is finished, in other words? It surely isn't just when it sends ... '</body></html>'

The CGI is finished when it exits. (Or, in rarer cases, when it closes STDOUT.) Some web servers are configured to time out if a script runs too long. ISPs do this for protection. It doesn't look like this is what you're running into.

The question that might you closer to your problem is

How does the client know when a CGI script is finished?
The answer here is that a client, which can be a browser or and LWP script, can't easily tell whether the page being fetched is being generated dynamically or not. (It's possible, but not easy.) But the client can detect that either (a) the number of bytes specified in the HTTP "Content-header" response have been read, or (b) that the server has closed the socket. What you're probably seeing is the result of the client timing out before (b) is satisified. "The web server hasn't sent me anything in the last n seconds, so it's probably borked."

A solution to this is to slowly dribble some content out to the client. Have your index.cgi disable buffering, emit some content immediately, then for each link you check that succeeds, emit that link immediately. This should keep the client from timing out.

In reply to Re: Perl Output/Web Server Question by dws
in thread Perl Output/Web Server Question by Cody Pendant

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.