in reply to LWP and Site Logins

An underused trick is to use LWP::Debug in your script. This will give you some indication as to what is going on during the request/response process.

As you've noted, your script returns a 302 header, which is a redirect. LWP::UserAgent will automatically follow redirects on GET and HEAD requests. Adding POST to that list will get you to the proper page.

Assuming you've entered a proper username and password you'll move on, other wise you'll be back at the login page.

use strict;
use warnings;

use HTTP::Cookies;
use LWP::UserAgent;
use LWP::Debug qw( + );

my $url      = 'https://noii.nasdaqtrader.com/';
my $agent    = LWP::UserAgent->new(
    cookie_jar            => HTTP::Cookies->new,
    requests_redirectable => [ 'GET', 'HEAD', 'POST' ]
);

my $response = $agent->post( $url, {
    txtUserName => '***',
    txtUserPass => '***'
} );

print $response->content;

[download]

--
"Go up to the next female stranger you see and tell her that her "body is a wonderland."
My hypothesis is that she’ll be too busy laughing at you to even bother slapping you." (src)

Replies are listed 'Best First'.
Re^2: LWP and Site Logins
by shagbark (Acolyte) on Jul 01, 2007 at 04:39 UTC
    I'm having strange behavior on several fronts.

    First, when I capture the actual web traffic, it's the same (almost) for both my browser, and for LWP, except that the packets captured by Ethereal when using the browser suddenly stop in the middle of the transmission. With LWP, you see the entire first web page; with the browser (Firefox 2), it ends like this: 

    function yadda(...
    // check for redirection
    return redirectCheck(pluginFound, redire

    [download]
    Consistently, it cuts off receiving after "redire" in Firefox, although it receives the entire TCP stream when using Internet Explorer. This is not a cache issue; I cleared the cache before trying with both browsers.

    Second, LWP handles the redirect automatically - as evidenced by the fact that, although the headers that Ethereal sees say 

    HTTP/1.1 302 Object Temporarily Moved
Connection: close
Date: Sun, 01 Jul 2007 03:53:25 GMT
Server: Microsoft-IIS/6.0
location: https://<...etc>

    [download]
    LWP's $response->status_line returns only "200 OK". However, that "200 OK" response is not captured by Ethereal. My program receives it, but Ethereal claims it's never been received over port 80. AND, LWP does not receive the redirected page in the response. Neither does the response contain the original page. The response that LWP saves is some third page, neither the original, nor the page redirected to.

    Third, the web browser continues on to display the next webpage, even though no more traffic was captured by Ethereal.

    The code looks like this: 

    my $ua = LWP::UserAgent->new(
   requests_redirectable => [ 'GET', 'HEAD', 'POST' ]
);
...
            $response = $ua->get($uri, @headers);

            # Handle redirects
            # This code never actually executes - LWP does it automati
+cally
            # That's why you never see the redirect message
            while ($response->is_redirect) {
               my $location = $response->header("location");
               print "  "x$level . "Redirected to $location\n";
               $response = $ua->get($location);
            }

            $page = $response->content;
            $success = $response->is_success;
            if (!$success) {
               print "LWP ERR: " . $response->status_line . "\n";
            }

    [download]
    Am I supposed to use LWP::Redirect? I find there is a module of that name, but no documentation mentions it.
[reply]
[d/l]
[select]

      It's really hard to tell exactly what's going on -- but...

      http is port 80, and https is port 443 -- this may explain some things if all you're doing is monitoring port 80 (though I've never used ethereal myself).

      If you don't want LWP to auto-follow redirects, change the requests_redirectable line to

      requests_redirectable => []

      [download]

      If you really do want to auto-follow redirects but it's not going to the same page, there may be some user-agent-based filtering happening on the server side (it's really hard to tell without seeing the actually output)

      Firefox's LiveHTTPHeades plugin is a really great way to see the request/response flow -- it may be a little easier to see what's going on with it instead.

      --
      "Go up to the next female stranger you see and tell her that her "body is a wonderland."
      My hypothesis is that she’ll be too busy laughing at you to even bother slapping you.      " (src)

[reply]
[d/l]

In Section Seekers of Perl Wisdom