Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to use LWP to create a small script that will go to a webpage, login, then do some other stuff. I can post to the site just fine, and I appear to be logged in. My problem is that the site redirects you to a different page after you are authenticated. My response object gives me the Location header with the file I should be going to. I can't figure out how to then move to this page. If I immediately try to do a GET on this page, I get taken back to the login page.

Replies are listed 'Best First'.
Re: LWP Posting login/redirect
by JediWizard (Deacon) on Oct 26, 2004 at 19:35 UTC

    My First instinct is that the login page probably sets a cookie once you've been authenticated. If this is the case you should make sure you have added a cookie jar to your user agent, otherwise subsequent requests will not include the cookie (which might make the site think you are not logged in). My other piece of advice would be to make sure the "POST" is in the requests_redirectable list. This will tell the user agent to automatically follow redirects even for POST requests (not the default behavior).See the below example:

    $ua = LWP::UserAgent->new; push @{ $ua->requests_redirectable }, 'POST'; $ua->cookie_jar(HTTP::Cookies->new(file => "C:/temp/lwpcookies.txt", a +utosave => 1));
    May the Force be with you
      I added a cookie, and that did the trick. It is now following the redirect and working great. Thanks JediWizard
      I already had the redirectable parameter set to include POST. I didn't think of the cookie though. I'll try that..thanks
Re: LWP Posting login/redirect
by elwarren (Priest) on Oct 27, 2004 at 00:08 UTC
    LWP will by default follow redirects between 10-15 times before giving up. It is a configurable parameter. Some of the more intelligent sites will also ignore you if your UserAgent isn't something well known like Mozilla or IE. Change the default UA string to avoid discrimination. Finally, some more intelligent (or not) sites will check your referer to try and avoid deep linking or (again) scripted webcrawlers.

    Lucky for us, all of these defenses can be overcome with LWP. Long live Perl! The swiss-army chainsaw!