LWP redirect problem, produces malformed URL?

FloydATC has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to write a tiny script to login and extract some information from our intranet server running eZ publish, by the use of LWP.

#!/usr/bin/perl

use strict;
use warnings;
use LWP;

my $path = "Infrastruktur/Svitsjoversikt";
my $loginurl = "http://intranett.oikt.local/user/login";
my $finaluri = "/Infrastruktur/Svitsjoversikt";
my $user = "foo";
my $pass = "bar";
my $param = "Login=$user&Password=$pass&LoginButton=1&RedirectURI=$fin
+aluri";

# Create a user agent
my $ua = LWP::UserAgent->new;
$ua->agent("monitorezex/1.0 ");
#push @{ $ua->requests_redirectable }, 'POST';   # Make POST requests 
+redirectable
$ua->cookie_jar({ file => "cookies.txt" });

# Create a request
my $req = HTTP::Request->new(
  POST => $loginurl
);
$req->content_type('application/x-www-form-urlencoded');
$req->content( $param );

# Pass request to the user agent and get a response back
my $res = $ua->request($req);

# Check the outcome of the response
if ($res->is_success) {
  print "SUCCESS!\n";
  print $res->as_string;
} else {
  print "FAILURE: " . $res->status_line . "\n";
  print $res->as_string;
}
exit;
[download]

The web server requires a username and password, upon successful authentication it responds with a 302 redirect to the requested URI, like so:

HTTP/1.1 302 Found
Cache-Control: no-cache, must-revalidate
Connection: close
Date: Fri, 09 Oct 2009 06:45:56 GMT
Pragma: no-cache
Location: http://intranett.oikt.local/Infrastruktur/Svitsjoversikt
Location: http://intranett.oikt.local/Infrastruktur/Svitsjoversikt
Server: Apache/1.3.39 (Win32) PHP/5.2.6
Content-Language: no-bokmaal
Content-Type: text/html; charset=utf-8
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Fri, 09 Oct 2009 06:45:56 GMT
Client-Date: Fri, 09 Oct 2009 06:45:56 GMT
Client-Peer: 10.10.8.120:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Refresh: 0;URL=http://intranett.oikt.local/Infrastruktur/Svitsjoversik
+t
Served-By: intranett.oikt.local
Set-Cookie: eZSESSIDoikt_intranett=v8igadrdc86koc225jk55or5e2; path=/
X-Powered-By: eZ Publish

<HTML><HEAD><META HTTP-EQUIV="Refresh" Content="0;URL=http://intranett
+.oikt.local/Infrastruktur/Svitsjoversikt"><META HTTP-EQUIV="Location"
+ Content="http://intranett.oikt.local/Infrastruktur/Svitsjoversikt"><
+/HEAD><BODY></BODY></HTML>
[download]

Notice that the login is successful, I get a cookie and a redirect URL. For some reason, if I uncomment the line to enable automatic redirection, the web server gets a malformed URL from my script:

10.15.0.2 - - [09/Oct/2009:08:38:31 +0200] "POST /user/login HTTP/1.1"
+ 302 246
10.15.0.2 - - [09/Oct/2009:08:38:32 +0200] "GET /Infrastruktur/Svitsjo
+versikt,%20http://intranett.oikt.local/Infrastruktur/Svitsjoversikt H
+TTP/1.1" 404 23015
[download]

I understand that this problem can be solved by fetching the new URL manually, but can someone point out why the automatic redirect doesn't work as I expect?

-- Time flies when you don't know what you're doing

Comment on LWP redirect problem, produces malformed URL? Select or Download Code

Replies are listed 'Best First'.
Re: LWP redirect problem, produces malformed URL? by gmargo (Hermit) on Oct 09, 2009 at 19:44 UTC
I believe this could be considered an LWP bug. It is triggered by the double "Location:" header. When LWP is processing the redirect request, it gets the new URI from the Location header with this bit of code: `# And then we update the URL based on the Location:-header. my $referral_uri = $response->header('Location');` [download] and since the "header()" method (in scalar mode) helpfully returns a comma separated list of all instances of the requested header, you end up with a weird URI. There are several possible solutions: Fix the server to only give one Location header (probably hard), or Fix LWP and distribute to world immediately (really hard), or Fiddle with the headers in the middle of the redirect process (easy). I would go with option #3. Create a new class based on LWP::UserAgent and override the redirect_ok() method. Here is some untested code that should do the trick: package FixRedirectLWPUserAgent; ########################################################## # Derived from LWP::UserAgent # Handle redirect_ok method to fix double Location header. ########################################################## use strict; use warnings; use LWP::UserAgent; our @ISA = qw( LWP::UserAgent ); sub redirect_ok { my ($self, $prospective_request, $response) = @_; # Check response for multiple location headers. my @locations = $response->header('Location'); if (scalar(@locations) > 1) { my $uri = $prospective_request->uri(); my @uris = split /, /,$uri; $prospective_request->uri($uris[0]) if scalar @uris; } # Call parent to do all the boring work. my $ok = $self->SUPER::redirect_ok($prospective_request, $response +); return $ok; } 1; [download]	[reply] [d/l] [select]
Re: LWP redirect problem, produces malformed URL? by Anonymous Monk on Oct 09, 2009 at 08:36 UTC
. For some reason, if I uncomment the line to enable automatic redirection, the web server gets a malformed URL from my script: Please show the LWP::Debug output ... then upgrade LWP :)	[reply]
Re^2: LWP redirect problem, produces malformed URL? by FloydATC (Deacon) on Oct 09, 2009 at 09:13 UTC
LWP::UserAgent::new: () LWP::UserAgent::request: () HTTP::Cookies::add_cookie_header: Checking intranett.oikt.local for co +okies HTTP::Cookies::add_cookie_header: Checking .oikt.local for cookies HTTP::Cookies::add_cookie_header: Checking oikt.local for cookies HTTP::Cookies::add_cookie_header: Checking .local for cookies LWP::UserAgent::send_request: POST http://intranett.oikt.local/user/lo +gin LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::Protocol::collect: read 178 bytes HTTP::Cookies::extract_cookies: Set cookie eZSESSIDoikt_intranett => k +ra0uraccka3vif17fut1mk445 LWP::UserAgent::request: Simple response: Found LWP::UserAgent::request: () HTTP::Cookies::add_cookie_header: Checking intranett.oikt.local for co +okies HTTP::Cookies::add_cookie_header: - checking cookie path=/ HTTP::Cookies::add_cookie_header: - checking cookie eZSESSIDoikt_intr +anett=kra0uraccka3vif17fut1mk445 HTTP::Cookies::add_cookie_header: it's a match HTTP::Cookies::add_cookie_header: Checking .oikt.local for cookies HTTP::Cookies::add_cookie_header: Checking oikt.local for cookies HTTP::Cookies::add_cookie_header: Checking .local for cookies LWP::UserAgent::send_request: GET http://intranett.oikt.local/,%20http +://intranett.oikt.local/ LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::Protocol::collect: read 570 bytes LWP::Protocol::collect: read 3072 bytes LWP::Protocol::collect: read 1016 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 1947 bytes LWP::UserAgent::request: Simple response: Not Found [download] But... This is libwww-perl-5.805. That's strange, I upgraded this from CPAN yesterday but something must have gone wrong. -- Time flies when you don't know what you're doing	[reply] [d/l]
Re: LWP redirect problem, produces malformed URL? by Gangabass (Vicar) on Oct 09, 2009 at 08:35 UTC
What is your LWP version? Also: it's strange that your servers reply with two similar Location: headers...	[reply]