Re^9: Sniffer::HTTP problem with timeout

Well, I've got another question.

Now that I've got my code working it would seem I am not doing something as far as deleting connections. I am seeing the memory usage continue to climb with time. I understand that a connection will use an amount of memory equal to it's total size but shouldn't that memory be released back to the pool once I write the file? Am I supposed to delete each connection in my callback after I save it?

I am really just using the example code with a couple lines to save the content as a file.

#!/usr/bin/perl
use strict;
use Net::Pcap;
use Sniffer::HTTP;
my $VERBOSE = 1;
  
  my $sniffer = Sniffer::HTTP->new(
  callbacks => {
      request  => sub { my ($req,$conn) = @_; print $req->uri,"\n" if 
+$req },
      response => sub { my ($res,$req,$conn) = @_; 
         print $req->uri,"\n" if ($req);
         return if (length $res->content == 0);
                (my $filename = $req->uri)=~s/.*\///;
        open (FILE,">",$filename);
        print FILE $res->content;
        close FILE;
        },
         
   log      => sub { print $_[0] if $VERBOSE },
   tcp_log  => sub { print $_[0] if $VERBOSE > 1 },
  },
  timeout => 1*60, # seconds after which a connection is considered st
+ale
   stale_connection => sub { my ($s,$conn,$key) = @_;
       my %test=%$conn;
       $conn->log->("$key is stale.");
       $s->remove_connection($key);
   },
);
   $sniffer->run('eth0'); # uses the "best" default device
[download]

I see how the stale_connection callback does this but the request and response callbacks do not get the information that stale_connection uses to delete the connection. Or am I supposed to be creating a new $sniffer->run() for each connection?

I have certainly missed something.

Thanks again for all your help.

Comment on Re^9: Sniffer::HTTP problem with timeout Download Code

Replies are listed 'Best First'.
Re^10: Sniffer::HTTP problem with timeout by Corion (Patriarch) on Mar 23, 2011 at 07:47 UTC
No, I think your usage is good, and matches the documentation. It is just that proper closing of connections seems to be handled badly in Sniffer::HTTP::Connection, in that a closed connection never signals to the main runloop that it got closed. In theory, you should see log messages like `Removed $host:$port` , but from looking at the code, that doesn't seem to happen. If you can provide me with a small, non-sensitive packet capture (via `dump-raw.pl`, included with Sniffer::HTTP) where a connection gets properly closed, that would be great. I can then test against this and even include it in the test suite so that the bug never comes back.	[reply] [d/l] [select]
Re^11: Sniffer::HTTP problem with timeout by ponley (Novice) on Mar 23, 2011 at 20:19 UTC
I see the `Removed $host:$port` occasionally but as you mentioned not nearly as often as I should. I would be happy to send you a capture, what do you consider to be small and how should I send it? Also I used cpan>install to install Sniffer::HTTP but I don't see dump-raw.pl in the /usr/local/share/perl/5.10.1/Sniffer folder. Do I need to get the full package or is it somewhere else? Update OK, locate is my friend, I found dump-raw.pl but I'm not sure what you want me to do with it. Do I use it to get a set of packets and then feed them to sniffer?	[reply] [d/l]
Re^12: Sniffer::HTTP problem with timeout by Corion (Patriarch) on Mar 23, 2011 at 20:57 UTC
`dump-raw.pl` lives at least in Sniffer::HTTP, in the bin directory. I will arrange that starting with the next release, it gets installed in the Perl directory as well. As for packet captures for the connection, I don't mind anything below 50MB for debugging. For keeping and distributing the test, I think that something below 1MB is better. My email address should be in the module too, but I can be reached via `corion@cpan.org`.	[reply] [d/l] [select]
Re^13: Sniffer::HTTP problem with timeout by ponley (Novice) on Mar 23, 2011 at 22:27 UTC