in reply to Re^2: Sniffer::HTTP problem with timeout
in thread Sniffer::HTTP problem with timeout

If the ->recv_buffer is empty, then the payload got already moved up to ->_response, at least that's what I read from my code in ->flush_received. I'm not sure how useful the data will be, but at least if you want to issue a second request using a Range: to resume downloading the data, it might be helpfull.

Replies are listed 'Best First'.
Re^4: Sniffer::HTTP problem with timeout
by ponley (Novice) on Mar 20, 2011 at 21:15 UTC
    Well.....After looking at, and trying your replies, Thank you very much for your help, It became clear that there was a random inconsistency in how things were running. I finally decided to run tcpdump and add a 'control' to the equation. suddenly everything worked just as it was supposed to. I did notice that tcpdump reported a packet dropped by the kernel though. I ran tcpdump a few more times and sometimes saw dropped packets and sometime not. I looked at the system resources in top and the system was never less than 50% idle, tcpdump was never more than 1% of cpu resources and I saw irq/18-b43 which while never over 4% cpu resources registered -51 for priority. I thought priorities ran from -20 - +20. I have no idea how to verify if the kernel is dropping packets when I am using your Sniffer::HTTP live but I am reasonably sure that is what my problem is.

    I guess what all this means is that is not anything in my code. So the question now is, How to stop the packet loss. I found some information regarding disk writes blocking tcpdumps ability to retrieve packets from the nic before the buffer overflows and a suggestion to increase buffer size with 
    echo 4194304 > /proc/sys/net/core/rmem_max; echo 4194304 > /proc/sys/n
+et/core/rmem_default

    [download]
    It didn't seem to help, do you have any idea that might help with the kernel packet loss and or what I can do about it. I am running this on a couple different P4 machines one being a 3.2G the other a dual core 2.4G so I wouldn't think horsepower is the issue. Is there something that I should be doing differently on my system so that pcap works better?

    Again, Thank you very much for helping me find the problem.
[reply]
[d/l]

      I'm sorry, but I have only very little knowledge of Linux kernels.

      In theory, TCP should be able to cope with the dropped packets and HTTP::Sniffer should be able to reconstruct the TCP stream despite the dropped packets. Maybe it is that the per-packet callback code of Net::Pcap blocks your kernel so long that it starts dropping packets, but as far as I'm aware, Net::Pcap runs asynchronously to the kernel. I'm not sure how you can easily verify that - maybe having sleep in the callback helps reproduce the dropped packets.

[reply]
        Well thank you again for helping me to locate the problem. I learned a few things in the process. I found something called Gulp that was written to address this problem by buffering but needs to be in the capture loop. I think that is beyond my ability so I will continue to look for an alternative solution.
[reply]

In Section Seekers of Perl Wisdom