Brak has asked for the wisdom of the Perl Monks concerning the following question:

I've got this script which uses LWP::UserAgent to connect to and receive data from a server which is not managed by me. It's managed by a department at my organization that doesn't really understand how to or doesn't want to fix minor internal issues. Therefore, I'm sort of stuck with having to deal with their... ignorance.

The Scenario
I have an initial URL. It's a fully qualified absolute URL like the documentation wants. However, the server 302-forwards the request to a relative URI, then gets mad because it's not an absolute URI.

The Question
How can I intercept the 302 request before it's sent back to LWP::UserAgent, so I can prepend the server's domain, which will make it a fully qualified absolute URI that the server won't bark at. If it's possible to not manually issue the redirect request, I'd be super happy.

The Response
So below is the output from Dumper() of my $ua User Agent object. I create some POST values, all basic ones, simple strings, and pass them to my initial URI and initiate the request to the server. The server responds with a 302 File moved response, pointing to a relative URI. (The initial request *must* be made to that URI and it *must* be forwarded to the secondary URI, as the first URI generates a session variable for the second one.) LWP picks this response up and re-issues the command to the new location, however, the server doesn't like the relative URL and wants an absolute URL instead. So it instead of returning the desired/intended response, it complains kicking back the 400 error "URL must be absolute". The process works as expected when run in a traditional web browser. I believe the reason it works at all is because the browser is actually fixing the URL from the Location header before issuing the redirect request. Probably doing this to be more tolerant of non-standard-compliant code.

### This is a dump of my LWP::UserAgent object ### ### It looks like the '_uri' has the incomplete relative ### URI in it. The script on the server is named 'f' and ### it has some query string args. ## Headers $VAR1 = bless( { 'client-warning' => 'Internal response', 'client-date' => 'Mon, 09 Nov 2009 20:43:38 GMT', 'content-type' => 'text/plain' }, 'HTTP::Headers' ); ## Content $VAR1 = '400 URL must be absolute '; $VAR1 = bless( { '_content' => 'p_v03=AUACB&p_flow_id=145&p_flow_step_ +id=12&p_request=GO&p_t04=&p_instance=3135116688739984&p_md5_checksum= +&p_arg_names=2162829369140202&p_t01=CF&p_t02=2010', '_uri' => bless( do{\(my $o = 'f?p=145:12:31351166887 +39984:::::')}, 'URI::_generic' ), '_headers' => bless( { 'user-agent' => 'MyOrganizatio +nHere/WebGrabber (ScreptNameHere / Perl 5.008008)', 'content-type' => 'application +/x-www-form-urlencoded', 'content-length' => 156, 'authorization' => 'Basic AnAU +THENTICATIONhashHERE ' }, 'HTTP::Headers' ), '_method' => 'POST' }, 'HTTP::Request' ); $VAR2 = bless( { '_content' => '400 URL must be absolute ', '_rc' => 400, '_headers' => bless( { 'client-warning' => 'Internal +response', 'client-date' => 'Mon, 09 Nov +2009 20:43:38 GMT', 'content-type' => 'text/plain' }, 'HTTP::Headers' ), '_msg' => 'URL must be absolute', '_request' => $VAR1 }, 'HTTP::Response' );
Any advice anyone can provide would be greatly appreciated. Thanks.

Replies are listed 'Best First'.
Re: LWP::UserAgent and correcting automatic redirection.
by Corion (Patriarch) on Nov 09, 2009 at 22:19 UTC

    Just hook your code into the response_done or response_redirect handler, and modify the response there. See LWP::UserAgent.

Re: LWP::UserAgent and correcting automatic redirection.
by gmargo (Hermit) on Nov 09, 2009 at 22:32 UTC

      Thanks for the tip gmargo, I installed your bit of code and it does seem to do what it's intended to do. Even though your code would have helped in the case of the server returning multiple Location headers, which is different from my problem, it did get me a head-start into verifying that my problem is different from what I originally thought it was. I now believe that the server is returning the CORRECT Location headers, which includes 2x 302 forwarding jumps, but I'm not really sure where to go from here since it's reporting "400 URL must be absolute". Any thoughts on general problem solving techniques I can try, things to check, ways to profile and get some information back as to why their server may be responding that way?

        Since you don't post any traffic except the final error, it's difficult to say much about your particular problem. In various programs I've written, a similar problem comes up, where the URI must be extended from relative to absolute with a "base" host or uri derived from the request.

        I don't know what your debugging environment is, but as a "general problem solving technique" for LWP::UserAgent, one of the most valuable tools would probably be a proxy in the middle to record all traffic between the application and the server.

        There are two methods I frequently use for logging traffic - one is another form of an LWP::UserAgent derived class that logs everything sent to/from the server. Another is an external java tool called "paros" http://www.parosproxy.org. Once you can see all the two-way traffic, it may become more obvious when you need to edit the URI on the fly.