in reply to Extract chunked/gzip data from pcap file

use PerlIO::gzip;
  • Comment on Re: Extract chunked/gzip data from pcap file

Replies are listed 'Best First'.
Re^2: Extract chunked/gzip data from pcap file
by oakb (Scribe) on Dec 28, 2009 at 14:17 UTC

    I have the PerlIO::gzip module installed, and I've tried using it. However, there's more to the puzzle than just using that module.

    When I look at these packets in Wireshark, a packet can be viewed in three different forms: (1) full frame, with the binary data intact and still encoded; (2) "De-chunked entity body", which exhibits the majority of the binary data intact -- but which has removed the "chunked encapsulation" (for lack of a better term); and (3) "Uncompressed entity body", which shows no binary data, just the decoded HTML text.

    This leads me to believe that there is some intermediate step that is required to remove the "chunked encapsulation", before being able to hand the "clean" compressed data to PerlIO::gzip as a stream for decompression.

      Most likely, you will need to reassemble the TCP frames into the complete TCP message, then parse the HTTP response from that, and then decode the payload of the HTTP response. This is something I've done with Sniffer::HTTP, which gices you a HTTP::Response object for each completed request. You can then query the ->decoded_content method of HTTP::Response to get the uncompressed data out.

      Note that I have at least one report of Sniffer::HTTP having a memory leak, so be sure to test the memory requirements before rolling it out on a large scale. Unfortunately, Net::Pcap doesn't currently build for me on Win32, so I can't conveniently replicate the environment to actually test things.

        I can't get Sniffer::HTTP to install on Win32 ActivePerl. From the sound of it, I might be better off not installing it anyway. Of course, I have HTTP::Response as part of my default installation. Is there any way to create an HTTP::Response object without Sniffer::HTTP?