LWP::UserAgent hacking

DaTa! has asked for the wisdom of the Perl Monks concerning the following question:

I want LWP::UserAgent to log the following information:

requested uri, time it took to fetch, content length and header length.

The following code works unless somebody uses :content_cb, in this case $content_length is zero, because the content is not included in the HTTP::Response object. Has somebody an idea to get the content-length anyway?

Here is my code.

package LWP::UserAgent::Trace;

use Time::HiRes;
use Data::Dumper;
use base qw/LWP::UserAgent/;

sub send_request {
  my($self, $request, $arg, $size) = @_;

  my $t0 = [Time::HiRes::gettimeofday];
  my $response = $self->SUPER::send_request($request,$arg,$size);
  my $elapsed = Time::HiRes::tv_interval($t0); 

  my $content_length = length($response->content);
  my $header_length = length($response->headers_as_string);
  print STDERR sprintf "%s %f %d %d %d\n",
    $request->uri, $elapsed, $content_length,
    $header_length, $content_length+$header_length;

  return $response;
}

1;
[download]

Comment on LWP::UserAgent hacking Download Code

Replies are listed 'Best First'.
Re: LWP::UserAgent hacking by Thelonius (Priest) on Mar 04, 2006 at 19:57 UTC
Here's a method that will work for chunked responses, too: sub send_request { my($self, $request, $arg, $size) = @_; my $content_length = 0; if (ref $arg && ref $arg eq "CODE") { my $original_arg = $arg; $arg = sub { $content_length += length($_[0]); return &$original_arg(@_); }; } my $t0 = [Time::HiRes::gettimeofday]; my $response = $self->SUPER::send_request($request,$arg,$size); my $elapsed = Time::HiRes::tv_interval($t0); $content_length = length($response->content) unless $content_length; my $header_length = length($response->headers_as_string); print STDERR sprintf "%s %f %d %d %d\n", $request->uri, $elapsed, $content_length, $header_length, $content_length+$header_length; return $response; } [download]	[reply] [d/l]
Re: LWP::UserAgent hacking by Thelonius (Priest) on Mar 04, 2006 at 17:10 UTC
As long as LWP doesn't support chunked responses (I think it still doesn't), this will work: `my $content_length = $response->content_length;` [download]	[reply] [d/l]