in reply to uncompress gzip data in a callback

In essence you can't decompress zipped data until you have the whole lot. It (they, there are many 'zip' implementations) simply aren't designed to work that way.

If the trade off of zipping/unzipping against bandwidth is essential, then you have little choice but to forget the callback and wait until you've received the whole response bofore you unzip and display.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^2: uncompress gzip data in a callback
by Weevil (Novice) on May 04, 2010 at 09:52 UTC

    Actually you can - the DEFLATE algorithm encodes data as a stream of blocks which can be decoded sequentially, as zcat does. Gzip wraps a DEFLATE stream with a header and footer but since the footer is for integrity checking it is not useful until after the whole stream has been decompressed.

    If you're specifically talking about PKZIP you're correct, but that's because PKZIP is an archive format containing multiple files each compressed with DEFLATE, and thus is not the same thing.

[reply]

      The problem is, the :content_cb callback gets called with odd sized chunks of data as they are received, and if you pass those to inflate(), you just get ' data error'.

      I've tried accumulating the odd sized chunks as they arrive, and passing the buffer to inflate() as they accumulate, but it has never actually decoded anything until the entire request has completed.

      And if you have to accumulate the entire content before decoding it, you might as well skip the callback and let LWP decode it for you.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        That sounds like you aren't maintaining the context of the compressed data stream between invocations of inflate. The code I posted earlier today shows how to handle that.
[reply]

      You're right!

      Once you match the WindowBits on the inflator, to the compressor used by the server, and don't try to do the buffering yourself, Compress::Raw::Zlib works perfectly.

      Many thanks. I've wanted this for ages.

[reply]

In Section Seekers of Perl Wisdom