Re: Determining Content-Length when there is no Content-Length header

You can try to use a partial GET (byte range) to seek around the file to find its end by trial (and avoid downloading 20GB), but I'm pretty sure 99% of the resources supporting a partial GET will also report a Content-Length in the header.

Short answer: no, AFAIK

Update:Here's what I found in the standard. I don't quite understand the verbiage, but take a look at this.

Comment on Re: Determining Content-Length when there is no Content-Length header

Replies are listed 'Best First'.
Re^2: Determining Content-Length when there is no Content-Length header by erroneousBollock (Curate) on Sep 30, 2007 at 03:25 UTC
You can try to use a partial GET (byte range) to seek around the file to find its end by trial (and avoid downloading 20GB), but I'm pretty sure 99% of the resources supporting a partial GET will also report a Content-Length in the header. They're definitely not required to be present together. Furthermore, the server doesn't necessarily have to tell you that it doesn't support range requests at all (let alone in a useful manner). The RFC says: Note: clients cannot depend on servers to send a 416 (Requested range not satisfiable) response instead of a 200 (OK) response for an unsatisfiable Range request-header, since not all servers implement this request-header. The definitive answer is no. Now, for hacker... Alternatively, why not use the ':content_cb' callback of LWP::UserAgent. With that callback, you can implement your own semantics for max-content-length; if you decide you no-longer want to fetch the file when you reach say 2k, just abort the request by die()ing. -David	[reply]

Replies are listed 'Best First'.

Re^2: Determining Content-Length when there is no Content-Length header
by erroneousBollock (Curate) on Sep 30, 2007 at 03:25 UTC

You can try to use a partial GET (byte range) to seek around the file to find its end by trial (and avoid downloading 20GB), but I'm pretty sure 99% of the resources supporting a partial GET will also report a Content-Length in the header.

Furthermore, the server doesn't necessarily have to tell you that it doesn't support range requests at all (let alone in a useful manner).
The RFC says:

Note: clients cannot depend on servers to send a 416 (Requested range not satisfiable) response instead of a 200 (OK) response for an unsatisfiable Range request-header, since not all servers implement this request-header.

Now, for hacker...

Alternatively, why not use the ':content_cb' callback of LWP::UserAgent. With that callback, you can implement your own semantics for max-content-length; if you decide you no-longer want to fetch the file when you reach say 2k, just abort the request by die()ing.

-David

[reply]