User-Agent in LWP::UserAgent

msel has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I am trying to rewrite the User-Agent header using the request_preprepare handler. For testing purposes, I have a webserver set up that redirects / to /dog, /dog to /cat, and /cat to /cat/.

I have the following code:

use strict;
use warnings;

use LWP::UserAgent;

my $url = "http://127.0.0.1";
my $ctr = 0;

my $ua = LWP::UserAgent->new();
$ua->agent('BOGUS');

$ua->add_handler( request_preprepare => sub { $ua->agent("TEST $ctr");
+ print "User-Agent: ", $ua->agent, "\n"; $ctr++; });

my $response = $ua->get($url);
[download]

Running the above results in the following output:

User-Agent: TEST 0
User-Agent: TEST 1
User-Agent: TEST 2
User-Agent: TEST 3

But the webserver log show the following:

127.0.0.1 - - 12/Aug/2014:23:06:09 -0400 "GET / HTTP/1.1" 302 204 "-" "TEST 0"
127.0.0.1 - - 12/Aug/2014:23:06:09 -0400 "GET /dog HTTP/1.1" 302 204 "-" "TEST 0"
127.0.0.1 - - 12/Aug/2014:23:06:09 -0400 "GET /cat HTTP/1.1" 302 205 "-" "TEST 0"
127.0.0.1 - - 12/Aug/2014:23:06:09 -0400 "GET /cat/ HTTP/1.1" 200 669 "-" "TEST 0"

It sets "TEST 0" as the initial User-Agent for the request for / (correctly replacing "BOGUS"), but it never increments for /dog, /cat, or /cat/ although it appears the handler is being called.

What am I doing wrong?

I can make this work by setting max_redirects to 0 and doing the redirect work manually by creating new LWP::UserAgent objects, but that is less elegant.

Thanks!

Comment on User-Agent in LWP::UserAgent Download Code

Replies are listed 'Best First'.
Re: User-Agent in LWP::UserAgent by Haarg (Priest) on Aug 13, 2014 at 06:47 UTC
Internally, ->agent is controlling the default header User-Agent. Default headers effect the creation of new requests, which is why your first request takes the adjusted agent string. When following redirects, the previous request is cloned and then adjusted. No default headers are applied. The request_preprepare handler is still run for the cloned requests. It is given the request as its first parameter. Changing the User-Agent header in that handler should give you the results you expected. `$ua->add_handler( request_preprepare => sub { my ($req) = @_; $req->header('User-Agent', "TEST $ctr"); print "User-Agent: ", $req->header('User-Agent'), "\n"; $ctr++; }, );` [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re: User-Agent in LWP::UserAgent
by Haarg (Priest) on Aug 13, 2014 at 06:47 UTC

Internally, ->agent is controlling the default header User-Agent. Default headers effect the creation of new requests, which is why your first request takes the adjusted agent string. When following redirects, the previous request is cloned and then adjusted. No default headers are applied.

The request_preprepare handler is still run for the cloned requests. It is given the request as its first parameter. Changing the User-Agent header in that handler should give you the results you expected.

$ua->add_handler(
  request_preprepare => sub {
    my ($req) = @_;
    $req->header('User-Agent', "TEST $ctr");
    print "User-Agent: ", $req->header('User-Agent'), "\n";
    $ctr++;
  },
);
[download]

[reply]
[d/l]