I manage many clients' firewalls, and regularly generate pcap packet trace files of traffic passing through these firewalls. I do a lot of searching, matching, and extracting of data from these files, and often use Net::TcpDumpLog to automate the process.

I find myself stymied, however, by HTML that has been optimized for download speed through the use of chunked Transfer-Encoding and gzip Content-Encoding. Since the text HTML has been turned into binary data, I can't automate the parsing process and systematically extract interesting information.

Is there a relatively simple way to decompress and decode this data so that it can be manipulated automatically in my program?

Here is what I have so far:

#!/usr/bin/perl


use strict;
use Net::TcpDumpLog;

my $log                        =  Net::TcpDumpLog->new();
                                  $log->read( "/my/tracefile.pcap" );
my $maxindex                   =  $log->maxindex();
my $gzip                       =  0;

foreach my $index ( 0..$maxindex ) {
    my ( $length_orig, $length_incl, $drops, $seconds, $milliseconds )
+  =  $log->header( $index );
    my $data                   =  $log->data( $index );
    
    if (( $data  =~ /Transfer-Encoding: chunked/g ) && ( $data  =~ /Co
+ntent-Encoding: gzip/g )) {
        $gzip++;
        print $index + 1 . "\t$length_orig\t$length_incl\t$seconds\t$m
+illiseconds\n";
        print "\t$data\n\n";
    }
}

print "$gzip chunked-gzip packets.";

[download]

In reply to Extract chunked/gzip data from pcap file by oakb

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.