in reply to LWP hidden 64k content line length limit

Are you really certain that the problem is with wget and LWP, and not maybe with the web server sending different things according to (for example) the User-Agent header?

As HTTP also can transfer binary data without "lines" at all, I highly doubt that there is a "64k line length limit" in either wget or LWP.

I would investigate the problem by making certain that wget, LWP and your Windows browsers send the identical headers to the remote end and inspecting closely what gets sent back.

  • Comment on Re: LWP hidden 64k content line length limit

Replies are listed 'Best First'.
Re^2: LWP hidden 64k content line length limit
by VZmaster (Initiate) on Jan 08, 2016 at 10:15 UTC

    I think you are right about web server relativity, when i created a copy of the page on my web server wget downloaded it fine.

    So it's seems wget/LWP can't handle something related to line length from original server

    Page is php generated, so headers doesnt include page length information, so this is why there is no error when page interrups.
[reply]

      So it's seems wget/LWP can't handle something related to line length from original server

      Which version of LWP?

[reply]
Re^2: LWP hidden 64k content line length limit
by VZmaster (Initiate) on Jan 09, 2016 at 07:54 UTC
    Investigation headers pointed at gzip compression, all browsers getting content gzipped while wget and lwp gettin it text/plain.

    I disabled gzip in browsers and whola truncated pages. So problem origins lies on the server side.

    But adding header Accept-Encoding:gzip, deflate, sdch not helping in wget, still getting text/plain. I've copied exact header from browser, but anyway it getting text/plain response.

    How to get gzipped response in lwp?

[reply]
[d/l]
      send all the same headers
[reply]
        Tried wget, not working. I'll try in LWP also.
[reply]

In Section Seekers of Perl Wisdom