in reply to Re^7: Sniffer::HTTP problem with timeout
in thread Sniffer::HTTP problem with timeout

No, thank you for reporting back how you solved your problem. This means that I'll update Sniffer::HTTP to make the snapshot length configurable from the outside.

My vague guess, not based on any evidence is that maybe libpcap allocates a fresh buffer or keeps a (too small) pool of snaplen byte buffers for each call to its capture loop. If we call the pcap loop quick enough, every buffer will only ever hold one packet, but maybe there are not enough buffers in the pool to store the packets that come in in the meantime. Alternatively, maybe I misunderstand libpcap, and each buffer is only intended for one packet, and it doesn't keep a pool of buffers around. Then the buffer of 128k is vastly oversized anyway.

But that's all speculation - my approach will be to drop the default buffer size to 16k and to make the buffer size easily configurable from the outside when constructing the capture or calling ->run.

Update: Released as 0.20 onto CPAN. Upon rereading Net::Pcap, it seems that the snaplen is per-packet, so the generous size of 128000 should have led to errors much earlier.

Replies are listed 'Best First'.
Re^9: Sniffer::HTTP problem with timeout
by ponley (Novice) on Mar 22, 2011 at 23:27 UTC
    Well, I've got another question.

    Now that I've got my code working it would seem I am not doing something as far as deleting connections. I am seeing the memory usage continue to climb with time. I understand that a connection will use an amount of memory equal to it's total size but shouldn't that memory be released back to the pool once I write the file? Am I supposed to delete each connection in my callback after I save it?

    I am really just using the example code with a couple lines to save the content as a file. 

    #!/usr/bin/perl
use strict;
use Net::Pcap;
use Sniffer::HTTP;
my $VERBOSE = 1;
  
  my $sniffer = Sniffer::HTTP->new(
  callbacks => {
      request  => sub { my ($req,$conn) = @_; print $req->uri,"\n" if 
+$req },
      response => sub { my ($res,$req,$conn) = @_; 
         print $req->uri,"\n" if ($req);
         return if (length $res->content == 0);
                (my $filename = $req->uri)=~s/.*\///;
        open (FILE,">",$filename);
        print FILE $res->content;
        close FILE;
        },
         
   log      => sub { print $_[0] if $VERBOSE },
   tcp_log  => sub { print $_[0] if $VERBOSE > 1 },
  },
  timeout => 1*60, # seconds after which a connection is considered st
+ale
   stale_connection => sub { my ($s,$conn,$key) = @_;
       my %test=%$conn;
       $conn->log->("$key is stale.");
       $s->remove_connection($key);
   },
);
   $sniffer->run('eth0'); # uses the "best" default device

    [download]


    I see how the stale_connection callback does this but the request and response callbacks do not get the information that stale_connection uses to delete the connection. Or am I supposed to be creating a new $sniffer->run() for each connection?

    I have certainly missed something.

    Thanks again for all your help.
[reply]
[d/l]

      No, I think your usage is good, and matches the documentation. It is just that proper closing of connections seems to be handled badly in Sniffer::HTTP::Connection, in that a closed connection never signals to the main runloop that it got closed. In theory, you should see log messages like Removed $host:$port , but from looking at the code, that doesn't seem to happen.

      If you can provide me with a small, non-sensitive packet capture (via dump-raw.pl, included with Sniffer::HTTP) where a connection gets properly closed, that would be great. I can then test against this and even include it in the test suite so that the bug never comes back.

[reply]
[d/l]
[select]
        I see the Removed $host:$port occasionally but as you mentioned not nearly as often as I should. I would be happy to send you a capture, what do you consider to be small and how should I send it? Also I used cpan>install to install Sniffer::HTTP but I don't see dump-raw.pl in the /usr/local/share/perl/5.10.1/Sniffer folder. Do I need to get the full package or is it somewhere else?

        Update

        OK, locate is my friend, I found dump-raw.pl but I'm not sure what you want me to do with it. Do I use it to get a set of packets and then feed them to sniffer?
[reply]
[d/l]

In Section Seekers of Perl Wisdom