in reply to Sniffer::HTTP problem with timeout

The ->recv_buffer holds the data that was captured. So in your stale_connection callback, you can use it to retrieve what was received so far.

Replies are listed 'Best First'.
Re^2: Sniffer::HTTP problem with timeout
by ponley (Novice) on Mar 20, 2011 at 09:25 UTC
    Thanks for the quick reply. That's what I had thought. I have this for the stale_connection callback. 
    stale_connection => sub { 
   my ($s,$conn,$key) = @_;
   print "Stale Connection\n".${$conn->recv_buffer}."\n";
   $conn->log->("$key is stale.");
   $s->remove_connection($key);
},

    [download]
    and all I get is a blank line following the Stale Connection text. After reading your response I went into Sniffer/Connection/HTTP.pm and added 
    print ${$self->recv_buffer};

    [download]
    which does print the data. So somehow I seem to be losing the data on the way back to my code. What am I doing wrong?
[reply]
[d/l]
[select]

      If the ->recv_buffer is empty, then the payload got already moved up to ->_response, at least that's what I read from my code in ->flush_received. I'm not sure how useful the data will be, but at least if you want to issue a second request using a Range: to resume downloading the data, it might be helpfull.

[reply]
[d/l]
[select]
        Well.....After looking at, and trying your replies, Thank you very much for your help, It became clear that there was a random inconsistency in how things were running. I finally decided to run tcpdump and add a 'control' to the equation. suddenly everything worked just as it was supposed to. I did notice that tcpdump reported a packet dropped by the kernel though. I ran tcpdump a few more times and sometimes saw dropped packets and sometime not. I looked at the system resources in top and the system was never less than 50% idle, tcpdump was never more than 1% of cpu resources and I saw irq/18-b43 which while never over 4% cpu resources registered -51 for priority. I thought priorities ran from -20 - +20. I have no idea how to verify if the kernel is dropping packets when I am using your Sniffer::HTTP live but I am reasonably sure that is what my problem is.

        I guess what all this means is that is not anything in my code. So the question now is, How to stop the packet loss. I found some information regarding disk writes blocking tcpdumps ability to retrieve packets from the nic before the buffer overflows and a suggestion to increase buffer size with 
        echo 4194304 > /proc/sys/net/core/rmem_max; echo 4194304 > /proc/sys/n
+et/core/rmem_default

        [download]
        It didn't seem to help, do you have any idea that might help with the kernel packet loss and or what I can do about it. I am running this on a couple different P4 machines one being a 3.2G the other a dual core 2.4G so I wouldn't think horsepower is the issue. Is there something that I should be doing differently on my system so that pcap works better?

        Again, Thank you very much for helping me find the problem.
[reply]
[d/l]

In Section Seekers of Perl Wisdom