Elliott has asked for the wisdom of the Perl Monks concerning the following question:

I am using LWP:Simple to read an HTML template, replace a few lines with personalised stuff and re-serve it. This is a technique I have used often before without problem...

Today, I have done this and the page does not appear to complete. In fact, if you stop it and look at the source, everything is there. But if you don't stop it, the Explorer or Netscape logos carry on animating.

This has often irritated me on other people's sites. Now I'm guilty too! Anyone got any idea what I'm doing wrong?

Replies are listed 'Best First'.
Re: Page doesn't appear to finish
by dws (Chancellor) on Nov 25, 2002 at 03:22 UTC
    I have done this and the page does not appear to complete.

    Depending on how you've got this coded (which we can only speculate about, since you haven't shown us code), you may be suffering from the HTTP/1.1 "Connection" optimization. If the HTTP header of request you're sending includes   Connection: keep-alive then the site you're fetching the template from has kept the socket open, and is quietly waiting for your next request. If your script is doing something like 

      while ( $chunk = <$socket> ) {
      print $chunk;
  }

    [download]
    then you'll hang until the web server times out and closes the socket.

    If this is the problem, sending   Connection: close with the request tells the web server to close the connection after serving you the template.

    Alternatively, you can honor the Content-length: response header, read exactly that many bytes of response, then close the socket yourself.

[reply]
[d/l]
[select]
Re: Page doesn't appear to finish
by pg (Canon) on Nov 25, 2002 at 07:12 UTC
    Usually the problem is the Content-Length field. I had the same kind of thing before. Most of the client side program determines whether it has received the whole page, base on the value received in Content-Length field.

    Better to fix the Content-Length field when you modify the content of a page.

    But, some time you don't need to fix the length, it still appears to be okay, why? That is because, the client reads in msg from socket chunk by chunk, not bytes by bytes, so there is a grey area. Look at the folllowing two examples:

    • (Here is a negative case) The Content-Length = 1001, when the real content is only 1000 bytes long. Say, after received the last chunk of msg, the client received 1000 bytes, so it expects one more chunk containing the last byte, but the server has already transferred the whole content and stopped. The client will never receive any more bytes, and it just waits and waits until timeout.
    • (Here is a positive case, the grey area worked) The Content-Length = 999, when the real content is 1200 bytes long. Now after received the last chunk of msg, the client received 998 bytes, so it waits for the next chunk, the server trasfers the next chunk, which is 202 bytes long, and stops. The client has now received 1200 bytes, which is greater than 999, so it is also satisfied.
[reply]
      Brilliant! Thank you to both of you who told me about Content-length. Solved the problem instantly.

      I would like to understand more about "Connection: close". When/how would I send that? Presuambly it has to be in the headers i.e. before all content, but mightn't that close the connection too early?

[reply]
Re: Page doesn't appear to finish
by adrianh (Chancellor) on Nov 25, 2002 at 03:18 UTC

    Pure guess (since you didn't send us any code :-)

    Did you shorten the length of the HTML, and forget to adjust the Content-Length header?

[reply]

Back to Seekers of Perl Wisdom