inblosam has asked for the wisdom of the Perl Monks concerning the following question:

I can't seem to get the pages returned to me (to extract what I want to with regexp). I seem to only be getting a header or something (see code below). I am using the suggestions I received previously from this node (cacharbe's reply). I turned on LWP::Debug to see the headers and such, and I noticed the session ID extracted (or set?) in the second request is not the same as the first. Is that a problem? I don't get much of a result from just the login even (I was expecting the page itself). Thanks!
Here is the script:
#!/usr/lib/perl -w

use strict;

use LWP::UserAgent;
use Crypt::SSLeay;
use HTTP::Request::Common;
use HTTP::Cookies;
use LWP::Simple;
use LWP::Debug qw(+);

my $ua = LWP::UserAgent->new;

$ua->cookie_jar(HTTP::Cookies->new(file => "cookie_jar", autosave => 1
+));
   $ua->timeout(300);
   
my $req = POST 'https://adwords.google.com/select/main',
             ['login.userid'       => 'test@test.com', 
              'login.password'     => 'testabc',
              'cmd'             => 'LoginValidation',
              'login'            => 'Login'
             ];

    my $res = $ua->request($req);
 
    unless ($res->is_success) {
        print "Login Failed: : ". $res->status_line .  "\n";
        return 0;
    }

my $b = $req->as_string;

my $req = '';

print "\n\n\nThe 1st result is:\n$b\n";

my $req = HTTP::Request->new(GET=>'https://adwords.google.com/select/m
+ain?cmd=CampaignManagement&campaignid=0&timeperiod=simple&timeperiod.
+simpletimeperiod=today');
my $res = $ua->request($req);

my $c = $req->as_string;

print "\n\n\nThe 2nd result is:\n$c\n";

[download]


Here is the command line printout, with LWP::Debug on. 

use perl5::site_perl::5.005::LWP::UserAgent;    #perl2exe
use perl5::site_perl::5.005::HTTP::Request;    #perl2exe
use perl5::site_perl::5.005::HTTP::Message;    #perl2exe
use perl5::site_perl::5.005::HTTP::Headers;    #perl2exe
use perl5::site_perl::5.005::URI;    #perl2exe
use perl5::site_perl::5.005::URI::Escape;    #perl2exe
use /usr/libdata/perl/5.00503/overload.pm;    #perl2exe
use perl5::site_perl::5.005::HTTP::Response;    #perl2exe
use perl5::site_perl::5.005::HTTP::Status;    #perl2exe
use perl5::site_perl::5.005::HTTP::Date;    #perl2exe
use /usr/libdata/perl/5.00503/Time/Local.pm;    #perl2exe
use perl5::site_perl::5.005::LWP;    #perl2exe
use perl5::site_perl::5.005::LWP::Debug;    #perl2exe
use perl5::site_perl::5.005::LWP::Protocol;    #perl2exe
use perl5::site_perl::5.005::LWP::MemberMixin;    #perl2exe
use perl5::site_perl::5.005::i386-freebsd::Crypt::SSLeay;    #perl2exe
use perl5::site_perl::5.005::HTTP::Request::Common;    #perl2exe
use perl5::site_perl::5.005::HTTP::Cookies;    #perl2exe
use perl5::site_perl::5.005::HTTP::Headers::Util;    #perl2exe
use perl5::site_perl::5.005::LWP::Simple;    #perl2exe
LWP::UserAgent::new: ()
use perl5::site_perl::5.005::URI::https;    #perl2exe
use perl5::site_perl::5.005::URI::http;    #perl2exe
use perl5::site_perl::5.005::URI::_server;    #perl2exe
use perl5::site_perl::5.005::URI::_generic;    #perl2exe
use perl5::site_perl::5.005::URI::_query;    #perl2exe
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking adwords.google.com for cook
+ies
HTTP::Cookies::add_cookie_header: Checking .google.com for cookies
HTTP::Cookies::add_cookie_header: Checking google.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: POST https://adwords.google.com/select/m
+ain
LWP::UserAgent::_need_proxy: Not proxied
use perl5::site_perl::5.005::LWP::Protocol::https;    #perl2exe
use perl5::site_perl::5.005::LWP::Protocol::http;    #perl2exe
use perl5::site_perl::5.005::Net::HTTP;    #perl2exe
use /usr/libdata/perl/5.00503/mach/IO/Socket.pm;    #perl2exe
use /usr/libdata/perl/5.00503/mach/IO/Handle.pm;    #perl2exe
use perl5::site_perl::5.005::Net::HTTP::Methods;    #perl2exe
use perl5::site_perl::5.005::Net::HTTPS;    #perl2exe
use perl5::site_perl::5.005::i386-freebsd::Net::SSL;    #perl2exe
use perl5::site_perl::5.005::i386-freebsd::MIME::Base64;    #perl2exe
use /usr/libdata/perl/5.00503/integer.pm;    #perl2exe
use perl5::site_perl::5.005::URI::URL;    #perl2exe
use perl5::site_perl::5.005::URI::WithBase;    #perl2exe
LWP::Protocol::http::request: ()
use perl5::site_perl::5.005::i386-freebsd::Crypt::SSLeay::MainContext;
+    #perl2exe
use perl5::site_perl::5.005::i386-freebsd::Crypt::SSLeay::CTX;    #per
+l2exe
use perl5::site_perl::5.005::i386-freebsd::HTML::HeadParser;    #perl2
+exe
use perl5::site_perl::5.005::i386-freebsd::HTML::Entities;    #perl2ex
+e
use perl5::site_perl::5.005::i386-freebsd::HTML::Parser;    #perl2exe
LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 429 bytes
LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 255 bytes
HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => fzlsamv4y1
LWP::UserAgent::request: Simple response: OK



The 1st result is:
POST https://adwords.google.com/select/main
User-Agent: libwww-perl/5.63
Content-Length: 87
Content-Type: application/x-www-form-urlencoded

login.userid=test%40test.com&login.password=testabc&cmd=LoginValidatio
+n&login=Login

LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking adwords.google.com for cook
+ies
HTTP::Cookies::add_cookie_header: - checking cookie path=/select
HTTP::Cookies::add_cookie_header:  - checking cookie JSESSIONID=fzlsam
+v4y1
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header: Checking .google.com for cookies
HTTP::Cookies::add_cookie_header: Checking google.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: GET https://adwords.google.com/select/ma
+in?cmd=CampaignManagement&campaignid=0&timeperiod=simple&timeperiod.s
+impletimeperiod=today
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 429 bytes
LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 200 bytes
HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => h68javv4z1
LWP::UserAgent::request: Simple response: OK



The 2nd result is:
GET https://adwords.google.com/select/main?cmd=CampaignManagement&camp
+aignid=0&timeperiod=simple&timeperiod.simpletimeperiod=today
User-Agent: libwww-perl/5.63
Cookie: $Version=1; JSESSIONID=fzlsamv4y1; $Path="/select"

[download]

Thanks!

Michael Jensen
michael at inshift.com
http://www.inshift.com

Replies are listed 'Best First'.
Re: LWP, SSL, cookie session?
by u914 (Pilgrim) on Jun 18, 2002 at 14:10 UTC
    i think you are looking for your result in the wrong variable...

    You are printing out your request as a string.
    Probably what you want is the result, ie the returned page. The variable you're looking for in that case is: 

    $res->content;

    [download]
    Here's a modified version of your code, but be warned that i can't test it here at work (no perl, and i can't install it). 
    #!/usr/lib/perl -w

use strict;

use LWP::UserAgent;
use Crypt::SSLeay;
use HTTP::Request::Common;
use HTTP::Cookies;
use LWP::Simple;
use LWP::Debug qw(+);

my $ua = LWP::UserAgent->new;

$ua->cookie_jar(HTTP::Cookies->new(file => "cookie_jar", autosave => 1
+));
   $ua->timeout(300);
   
my $req = POST 'https://adwords.google.com/select/main',
             ['login.userid'       => 'test@test.com', 
              'login.password'     => 'testabc',
              'cmd'             => 'LoginValidation',
              'login'            => 'Login'
             ];

    my $res = $ua->request($req);
 
    unless ($res->is_success) {
        print "Login Failed: : ". $res->status_line .  "\n";
        return 0;
    }

#  These lines are unnecessary
#  my $b = $req->as_string;
#  my $req = '';

print "\n\n\nThe 1st result is:\n$res->content\n";

    [download]
    Or, if you really prefer to abstract, you could say: 
    my $b = $res->content;

    [download]
[reply]
[d/l]
[select]
Check Your Assumptions
by cacharbe (Curate) on Jun 18, 2002 at 14:19 UTC
    Well, you might want to change the lines that read:

    $var = $req->as_string;

    To

    $var = $res->as_string;

    Since, in the case of my example, $req is the request and $res is the response, and you want the text of the response, not the request.

    C-.

    Update: Start a response, get a phone call, get beat to the punch. *sigh*

[reply]
[d/l]
[select]
Re: LWP, SSL, cookie session?
by inblosam (Monk) on Jun 18, 2002 at 16:15 UTC
    Okay, now I am using res instead of req (I should have seen that). Now I get this message again (old node):
    Sorry, we're unable to process this request. There are two possible ca
+uses for this error: 

You accessed this page via a bookmark instead of entering through the 
+AdWords Select homepage 
You have disabled cookies in your browser.

    [download]

    My LWP::Debug gives me this about the cookies, if that helps:
    LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 429 bytes
LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 255 bytes
HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => 49d3aidt74
LWP::UserAgent::request: Simple response: OK
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking adwords.google.com for cook
+ies
HTTP::Cookies::add_cookie_header: - checking cookie path=/select
HTTP::Cookies::add_cookie_header:  - checking cookie JSESSIONID=49d3ai
+dt74
HTTP::Cookies::add_cookie_header:    it's a match
HTTP::Cookies::add_cookie_header: Checking .google.com for cookies
HTTP::Cookies::add_cookie_header: Checking google.com for cookies
HTTP::Cookies::add_cookie_header: Checking .com for cookies
LWP::UserAgent::send_request: GET https://adwords.google.com/select/ma
+in?cmd=CampaignManagement&campaignid=0&timeperiod=simple&timeperiod.s
+impletimeperiod=today
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 1019 bytes
LWP::Protocol::collect: read 1648 bytes
HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => 4iwdeedt77
LWP::UserAgent::request: Simple response: OK

    [download]

    The only thing I can see "unusual" is two different "JSESSIONID" id's, one being set and read, and the other one being set again.
    This is my latest code:

    #!/usr/lib/perl -w

use strict;

use LWP::UserAgent;
use Crypt::SSLeay;
use HTTP::Request::Common;
use HTTP::Cookies;
use LWP::Simple;
use LWP::Debug qw(+);

my $ua = LWP::UserAgent->new;

$ua->cookie_jar(HTTP::Cookies->new(file => "cookie_jar", autosave => 1
+));
$ua->timeout(500);
   
my $req = POST 'https://adwords.google.com/select/main',
             ['login.userid'       => 'test@test.com', 
              'login.password'     => 'testabc',
              'cmd'             => 'LoginValidation',
              'login'            => 'Login'
             ];

    my $res = $ua->request($req);
 
    unless ($res->is_success) {
        print "Login Failed: : ". $res->status_line .  "\n";
        return 0;
    }

my $b = $res->as_string;

print "\n\n\nThe 1st result is:\n$b\n";

my $req = HTTP::Request->new(GET=>'https://adwords.google.com/select/m
+ain?cmd=CampaignManagement&campaignid=0&timeperiod=simple&timeperiod.
+simpletimeperiod=today');
my $res = $ua->request($req);

my $c = $res->content;

print "\n\n\nThe 2nd result is:\n$c\n";

    [download]


    Michael Jensen
    michael at inshift.com
    http://www.inshift.com
[reply]
[d/l]
[select]
      Well, the issue is likely related to either cookies, or to the http_referer string....

      your error:
      Sorry, we're unable to process this request. There are two possible causes for this error:

      You accessed this page via a bookmark instead of entering through the AdWords Select homepage
      this one is probably because there is no http_referer set in the LWP au

      You have disabled cookies in your browser.

      this one is because the jsession id has changed

      The answer to these is probably to use LWP to get the page first, ie, establish a session... and then after that, return the form (jsession included) as it was presented.

      good luck!

[reply]
        Any ideas where I can find out how to implement that? I thought I was establishing a session by logging in (the login isn't passing any session ID along with it, is it? or is it part of a cookie set when just visiting the login page?). How do I "return the form as it was presented"? I am matching the exact login form, I thought. Thanks for the help!

        Michael Jensen
        michael at inshift.com
        http://www.inshift.com
[reply]

Back to Seekers of Perl Wisdom