SteveAyre has asked for the wisdom of the Perl Monks concerning the following question:

I need to measure how much bandwidth one of my perl scripts is using. It's downloading multiple pages from our web server via HTTPS using LWP.

Does anyone know how to get the number of bytes sent/received on the network by LWP?

Replies are listed 'Best First'.
Re: LWP send/received byte counters
by zentara (Cardinal) on Mar 09, 2009 at 13:52 UTC
    Yeah, there is a callback available to LWP that manually handles the data. It is used mostly for progress meters. When you use this callback, you MUST handle the filehandle yourself. You can google for "lwp progress', or here is a bare bones snippet:
    #!/usr/bin/perl -w use strict; use LWP::UserAgent; # don't buffer the prints to make the status update $| = 1; my $ua = LWP::UserAgent->new(); my $received_size = 0; my $url = 'http://www.cpan.org/authors/id/J/JG/JGOFF/parrot-0_0_7.tgz' +; print "Fetching $url\n"; my $request_time = time; my $last_update = 0; my $response = $ua->get($url, ':content_cb' => \&callback, ':read_size_hint' => 8192, ); print "\n"; sub callback { my ($data, $response, $protocol) = @_; my $total_size = $response->header('Content-Length') || 0; $received_size += length $data; # FIXME: write the $data to a filehandle or whatever should happen # with it here. my $time_now = time; # this to make the status only update once per second. return unless $time_now > $last_update or $received_size == $total_s +ize; $last_update = $time_now; print "\rReceived $received_size bytes"; printf " (%i%%)", (100/$total_size)*$received_size if $total_size; printf " %6.1f/bps", $received_size/(($time_now-$request_time)||1) if $received_size; }

    I'm not really a human, but I play one on earth My Petition to the Great Cosmic Conciousness

      That just counts the size of the body of the final response of a chain. It doesn't include any of the requests, the headers of any of the responses or the body of any but the last response.

        Yea, I was more interested in the raw amount sent and received on the network - including headers and after the encryption if possible.
Re: LWP send/received byte counters
by tirwhan (Abbot) on Mar 09, 2009 at 18:39 UTC

    If you're really looking for the raw network throughput, then you're probably best off sniffing the connections and tallying the packets and their payload manually. Take a look at Sniffer::HTTP and Net::Connection::Sniffer.


    All dogma is stupid.