kretch has asked for the wisdom of the Perl Monks concerning the following question:

Greetings all. I'm trying to log into a site (safeway.com) to get pull some nutritional information. I can get past the initial login phase, but fail when I try to get the following page. I suspect this has something to do with not using cookies properly. Could somebody please tell me what's wrong with the following piece of code? Thanks Kretch
$| = 1 ; use warnings ; use LWP::UserAgent; use HTTP::Cookies; use HTTP::Request::Common; use MD5; use strict; use Data::Dumper ; my $cookie_jar = HTTP::Cookies->new( file => $ENV{'HOME'}."/lwp_cookies.dat", autosave => 1, ); my $ua = LWP::UserAgent->new; $ua->proxy(['https','http','ftp'] => 'myinfo@myproxy') ; $ua->cookie_jar($cookie_jar) ; $ua->agent('Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.1 +1) Gecko/20071127 Firefox/2.0.0.11') ; my $response = $ua->post( 'https://shop.safeway.com/register/registernew.asp?signin +=1&returnTo=', [ "register" => '0', "rzipcode" => '90210', "zipcode" => '90210' ] ); die ("Error:", $response->status_line) unless $response->is_success ; $number = 142100398 ; my $url = "http://shop.safeway.com/dnet/RichProductInformation.aspx? +promo_windows=0&bpn=$number"; my $request = HTTP::Request->new('GET',$url); $request->protocol('HTTP/1.1') ; my $response2 = $ua->request($request) ; open (FILE,">$number".".html") || die ("cannot open file for write\n +"); print $response2->status_line."\n"; print FILE $response2->content ; close (FILE) ;

Replies are listed 'Best First'.
Re: Problems using LWP::UserAgent to log into safeway.com
by Gangabass (Vicar) on Jan 23, 2008 at 03:47 UTC

    Maybe it uses some Javascript magick to set additional cookies? Try to disable Javascript and get this page in your browser.

    You also can create request log in FireFox (using LiveHTTPHeaders).

      Yes, you nailed it. No javascript = no website load. I guess I'm out of luck then using Mechanize as well? Kretch

        Yes, you right. But you can look into Javascript code and simply implement same in Perl. Believe me it's not so hard :-).

Re: Problems using LWP::UserAgent to log into safeway.com
by Cody Pendant (Prior) on Jan 23, 2008 at 03:50 UTC
    Short answer, which you'll see a lot around here -- try again using WWW::Mechanize instead.


    Nobody says perl looks like line-noise any more
    kids today don't know what line-noise IS ...
      Will that work even for forms that are created/modified on-the-fly using javascript? That seems to be the case for that site (my deduction since the site doesn't work when JS is disabled) Kretch
        No, it won't. But it will handle all those nagging problems with cookies and so on, leaving you with only the JS problems to fix. And there's quite a bit on that in the FAQ.


        Nobody says perl looks like line-noise any more
        kids today don't know what line-noise IS ...