Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Perl LWP Can handle client-transfer-encoding = chunked encoding?

by FloydATC (Deacon)
on Jan 16, 2015 at 09:02 UTC ( [id://1113455]=note: print w/replies, xml ) Need Help??


in reply to Perl LWP Can handle client-transfer-encoding = chunked encoding?

Generally speaking, LWP does not care about the actual content or format, it simply passes on to the web server what kind of content encoding you are willing to accept, in the form of properly formatted request headers.

The web server should pay attention to this and either send data in an appropriate format/encoding (as indicated by the response headers) or inform you that it can't comply with your requirements. You may request documents in 9 bit Morse encoding compressed with a proprietary Javascript library but the server is free to say no. What the server should not do is send you data in an imaginary format when you clearly stated what you were willing to accept.

LWP has some convenience features that can help you with decoding of some common well-known encoding types but other than that you are free to take the binary content and decode it any way you like, which hopefully matches what the server indicated.

In this particular case, you may want to look at modules that deal with MIME content unless I'm very much mistaken.

-- FloydATC

Time flies when you don't know what you're doing

Replies are listed 'Best First'.
Re^2: Perl LWP Can handle client-transfer-encoding = chunked encoding?
by sam_bakki (Pilgrim) on Jan 16, 2015 at 09:38 UTC

    Hi FloydATC

    Thanks for the details. IMHO, I am suffering from the "chunk" mode of transfer. Basically, Sever sends the data in multiple chunks, So the LWP need to make a multiple requests to get full data. As you have stated, LWP might not care (or dont need to care) about the data and encoding / chunked or non chunked.

    Looks like Net::SSL can handle chunks properly where as IO::Socket::SSL is not handling it. I am not sure how to disable this chunking by using right http headers so server always sends data in one single chunk.

    Thanks & Regards,
    Bakkiaraj M
    My Perl Gtk2 technology demo project - http://code.google.com/p/saaral-soft-search-spider/ , contributions are welcome.

      No, that's usually not what "chunked" encoding means. It usually means that the response contains more than one file/document and you have to use the appropriate method to separate them.

      What you are referring to is a partial response, which the server is only allowed to send if you explicitly ask for it. This is commonly used when resuming a large download or seeking in streamed media. (See "range request".)

      -- FloydATC

      Time flies when you don't know what you're doing

        Transfer encoding 'chunked' was introduced with HTTP version 1.1 and is used when the length of the content is not known up-front, so that no 'Content-length' header can be given. Instead the content is sent in several chunks, each prefixed by the length of the chunk.
        It is not related to content-encoding like gzip and both can be used together. It is also not related to a content consisting of multiple parts. In this rarely used cases instead a Content-type of multipart/whatever is used and the client has to split the content by the given boundary, similar to multipart MIME messages in mails.
        Any HTTP client claiming support for version 1.1 (like LWP) must support this transfer encoding and there is no way to tell the server not to use it except switching to HTTP/1.0. The Accept-Encoding header only affects the accepted values for Content-Encoding (i.e. compression), not Transfer-Encoding.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1113455]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2024-03-29 07:08 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found