in reply to Re: Sniffer::HTTP problem with timeout
in thread Sniffer::HTTP problem with timeout

Thanks for the quick reply. That's what I had thought. I have this for the stale_connection callback. 
stale_connection => sub { 
   my ($s,$conn,$key) = @_;
   print "Stale Connection\n".${$conn->recv_buffer}."\n";
   $conn->log->("$key is stale.");
   $s->remove_connection($key);
},

[download]
and all I get is a blank line following the Stale Connection text. After reading your response I went into Sniffer/Connection/HTTP.pm and added 
print ${$self->recv_buffer};

[download]
which does print the data. So somehow I seem to be losing the data on the way back to my code. What am I doing wrong?

Replies are listed 'Best First'.
Re^3: Sniffer::HTTP problem with timeout
by Corion (Patriarch) on Mar 20, 2011 at 10:04 UTC

    If the ->recv_buffer is empty, then the payload got already moved up to ->_response, at least that's what I read from my code in ->flush_received. I'm not sure how useful the data will be, but at least if you want to issue a second request using a Range: to resume downloading the data, it might be helpfull.

[reply]
[d/l]
[select]
      Well.....After looking at, and trying your replies, Thank you very much for your help, It became clear that there was a random inconsistency in how things were running. I finally decided to run tcpdump and add a 'control' to the equation. suddenly everything worked just as it was supposed to. I did notice that tcpdump reported a packet dropped by the kernel though. I ran tcpdump a few more times and sometimes saw dropped packets and sometime not. I looked at the system resources in top and the system was never less than 50% idle, tcpdump was never more than 1% of cpu resources and I saw irq/18-b43 which while never over 4% cpu resources registered -51 for priority. I thought priorities ran from -20 - +20. I have no idea how to verify if the kernel is dropping packets when I am using your Sniffer::HTTP live but I am reasonably sure that is what my problem is.

      I guess what all this means is that is not anything in my code. So the question now is, How to stop the packet loss. I found some information regarding disk writes blocking tcpdumps ability to retrieve packets from the nic before the buffer overflows and a suggestion to increase buffer size with 
      echo 4194304 > /proc/sys/net/core/rmem_max; echo 4194304 > /proc/sys/n
+et/core/rmem_default

      [download]
      It didn't seem to help, do you have any idea that might help with the kernel packet loss and or what I can do about it. I am running this on a couple different P4 machines one being a 3.2G the other a dual core 2.4G so I wouldn't think horsepower is the issue. Is there something that I should be doing differently on my system so that pcap works better?

      Again, Thank you very much for helping me find the problem.
[reply]
[d/l]

        I'm sorry, but I have only very little knowledge of Linux kernels.

        In theory, TCP should be able to cope with the dropped packets and HTTP::Sniffer should be able to reconstruct the TCP stream despite the dropped packets. Maybe it is that the per-packet callback code of Net::Pcap blocks your kernel so long that it starts dropping packets, but as far as I'm aware, Net::Pcap runs asynchronously to the kernel. I'm not sure how you can easily verify that - maybe having sleep in the callback helps reproduce the dropped packets.

[reply]

In Section Seekers of Perl Wisdom