baghu has asked for the wisdom of the Perl Monks concerning the following question:

Hi

Please look at the headers of the Requests & responses of the redirected requests at the bottom.

I'm new to perl. I have issues logging into a https website.

When I submit the form through firefox browser, the url gets redirected to two other url's finally landing me into the homepage LOGGED IN. I see the cookies being updated very frequently by the web server though I've disabled java settings in the browser.

When I do submit the form using WWW::Mechanize, I end up in the homepage but NOT LOGGED IN. Looking at the redirected requests and response headers, I was able to figure out that the cookies get appended properly in the first redirected request. In the redirected response set-cookies I see the 'loggedin== true' and other user related info padded from the server. I assume I was logged into the website but on the 2nd auto redirected request, I barely see any info padded in the request from Mech's end. And the response to that request comes with 'loggedin==false' and lands me in the homepage.

Could anyone please help me understand things better.

Thanks

Below are the headers of the 3 Requests & Responses mentioned above. 


***************FOrm being submitted**************

*********** REQUEST SENT*********

POST https://secure.example.com/login/index/
Connection: keep-alive
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
+,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip
Accept-Language: en-us,en;q=0.5
Referer: https://secure.example.com/login/index/
TE:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1
+4) Gecko/2
0080404 Firefox/2.0.0.14
Content-Length: 54
Content-Type: application/x-www-form-urlencoded
Cookie: loggedin=true; example=.......................................
+...; uniq=
............................................;example_Tracking=......so
+urce...................medium..................term..................
+content..............
....campaign..................clid..................partnerid.........
+.........; account=true; user=..$USERNAME............................
+......................
......................................................................
+..........
......................................................;mpsession=.....
+..........
......................................................................
+..........
......................................................................
+..........
..........Source+Keyword..........................Source+Campaign.....
+..........
...........Source+Content..........................Referring+Domain...
+......none
.........Landing+Page.........login_index.........Landing+Page+Test...
+..........
.............Session+ID...............................................
+...Promo..
.......none.........User+Status.........Returning+User.........User+St
+atus+Detai
l.............................................................Visitor+
+Status....
.....Logged+In.........User+ID...................Account+Created......
+..........
Facebook+Status.......................................................
+..........
......................................................................
+..........
............................................Registration+Type.........
+..........
.......Registration+Method..........................Registration+Type-
+Method....
......................................................................
+.....; ste
ps=......; example_Tracking=..........................................
+..........
......................................................................
+..........
..........................................; uniq=.....................
+..........
...........................; steps=.......
Cookie2: $Version="1"
Keep-Alive: 300

usernameorid=$username&password=$password&rememberMe=on

************ RESPONSE*******************

HTTP/1.1 302 Found
Connection: close
Date: Fri, 13 Aug 2010 19:25:46 GMT
Location: /
Server: Apache/2.2.3 (Red Hat)
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 122
Content-Type: text/html; charset=UTF-8
Client-Date: Fri, 13 Aug 2010 19:25:46 GMT
Client-Peer: xx.xx.xxx.xxx:xxx
Client-Response-Num: 1
Client-SSL-Cert-Issuer: **xx**
Client-SSL-Cert-Subject: **xx**

Client-SSL-Cipher: RC4-SHA
Client-SSL-Warning: Peer certificate not verified
Set-Cookie: auth=.....$username.......................................
+.........
......................................................................
+.........
......................................................................
+.........
........................................; path=/; domain=.example.com;
+ httponly
Set-Cookie: user=.....$username.......................................
+.........
......................................................................
+.........
......................................................................
+........;
 expires=Fri, 20-Aug-2010 19:25:46 GMT; path=/; domain=.example.com; h
+ttponly
Set-Cookie: loggedin=true; path=/; domain=.example.com
Set-Cookie: mpsession=................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
..................Visitor+Status.........Logged+In.........User+ID....
+..........
..Account+Created.....................................................
+..........
.............Gender...................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
...................................................; path=/; domain=.e
+xample.com
Set-Cookie: steps=.......; expires=Fri, 20-Aug-2010 19:25:46 GMT; path
+=/; domain
=.example.com
X-Powered-By: PHP/5.3.2

\37\x8B\10\0\0\0\0\0\0\3%\x8DA\16\x830\14\4\xBF\22\xF9\x8E\34\xE8\21\x
+C7\xC7\xFE
#m\x8C\x82\4\tM\35$~_\xAB\\w4\xB3\x94u\xDF\x98\xB2\xC4\xC4\xB4\x8BF\x9
+7U\x8FA>}=
\x034Y\x9A|3\xB8w-*E\3\xF8\xB9\xB7-  \23\xDE\xD2\xAB\xA6\xCB\2#?\xFC\x
+E4\x9E\xB5
\x97ddd:\x98,f\x85\0\10\x8C\x84\xD1\24\33\xF1\26\xF0\x7F\xFC\3B\16\xB4
+>\x7F\0\0\
0

***********1st Redirected REQUEST SENT*********

GET https://secure.example.com/
Connection: keep-alive
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
+,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip
Accept-Language: en-us,en;q=0.5
Referer: https://secure.example.com/login/index/
TE:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1
+4) Gecko/2
0080404 Firefox/2.0.0.14
Cookie: loggedin=true; auth=...$username..............................
+..........
......................................................................
+..........
......................................................................
+..........
...........................................; example=.................
+..........
.............; uniq=............................................; exam
+ple_Tracki
ng=...................................................................
+..........
......................................................................
+..........
..................; account=true; user=....$username..................
+..........
......................................................................
+..........
......................................................................
+..........
......................; mpsession=....................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
........................................................; steps=......
+.; example
_Tracking=............................................................
+..........
......................................................................
+..........
........................; uniq=.......................................
+.........;
 steps=.......
Cookie2: $Version="1"
Keep-Alive: 300

(no content)

************ RESPONSE*******************

HTTP/1.1 302 Found
Connection: close
Date: Fri, 13 Aug 2010 19:25:41 GMT
Location: http://example.com/
Server: Apache/2.2.3 (Red Hat)
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 138
Content-Type: text/html; charset=UTF-8
Client-Date: Fri, 13 Aug 2010 19:25:47 GMT
Client-Peer: 69.20.109.107:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: **xx**
Client-SSL-Cert-Subject: **xx**

Client-SSL-Cipher: RC4-SHA
Client-SSL-Warning: Peer certificate not verified
Set-Cookie: mpsession=................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
..................Visitor+Status.........Logged+In.........User+ID....
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
.........................; path=/; domain=.example.com
Set-Cookie: steps=.....; expires=Fri, 20-Aug-2010 19:25:41 GMT; path=/
+; domain
=.example.com
X-Powered-By: PHP/5.3.2

\37\x8B\10\0\0\0\0\0\0\3m\xCEK\16\xC20\14\4\xD0\xABD\xD9\x83[\xD8\x81\
+xE3%\xF7H\
33C*\xE5S\x82\x83\xC4\xED1t\xDB\xADg\xE6\xC9\30%'\xC2\xC8>\20f\26o\xA2
+\xC8z\xE0g
_\xDE\xCE6\xBE7~Ek\xE6Z\x84\x8B8;\\{K\xEE\xD7\xB9\0L\xCB#\xB0O\xC7\xB9
+f\xB0@\10\
x1B3\xD5\xF0Qr\xA4\xF3p2\xB7\xDAK\xD0d$\\\t\x95W\xD3\xD9=\x80v\x8E\10^
+Y\35\xC2\x
86\xC2\xFF\xDD/.q31\xB5\0\0\0

***********2nd Redirected REQUEST SENT*********

GET http://example.com/
Connection: keep-alive
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9
+,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: gzip
Accept-Language: en-us,en;q=0.5
Referer: https://secure.example.com/login/index/
TE:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1
+4) Gecko/2
0080404 Firefox/2.0.0.14
Cookie: example_Tracking=.............................................
+..........
......................................................................
+..........
........................................; uniq=.......................
+..........
............; steps=......
Cookie2: $Version="1"
Keep-Alive: 300

(no content)

************ RESPONSE*******************

HTTP/1.1 200 OK
Connection: close
Date: Fri, 13 Aug 2010 19:25:48 GMT
Server: Apache/2.2.3 (Red Hat)
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 25265
Content-Type: text/html; charset=UTF-8
Client-Date: Fri, 13 Aug 2010 19:25:48 GMT
Client-Peer: 69.20.109.106:80
Client-Response-Num: 1
Set-Cookie: example=......................................; path=/; do
+main=.exam
ple.com; httponly
Set-Cookie: example_Tracking=.........................................
+..........
......................................................................
+..........
...........................................; expires=Fri, 20-Aug-2010 
+19:25:48 G
MT; path=/; domain=.example.com; httponly
Set-Cookie: example_redirect=false; expires=Fri, 13-Aug-2010 18:25:48 
+GMT; path=
/; domain=.example.com; httponly
Set-Cookie: loggedin=false; expires=Fri, 13-Aug-2010 18:25:48 GMT; pat
+h=/; dom
ain=.example.com; httponly
Set-Cookie: mpsession=................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
Session+ID............................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+..........
......................................................................
+.....; pat
h=/; domain=.example.com
Set-Cookie: steps=......; expires=Fri, 20-Aug-2010 19:25:48 GMT; path=
+/; domain
=.example.com
X-Powered-By: PHP/5.3.2

\37\x8B\10\0\0\0\0\0\0\3\xED\xBD{w\xDB6\xD2?\xFE\xF7\xB7\xAF\x82\xAB\x
+DD\xAD\x9D
_#\x89W\xDD\22\xBB\xC7\xB7$n\x9D\xC4O\xEC4\xDBg\xBB\xC7\x87\22!\x891Ej
+I\xCA\x8E\
xDA'\xEF\xFD7\0I](\14\xC1\x9B\xD2\xB4\xA7>m$\x91\x98\xC1\xE06\x98\xF9`
+\0<\xFF\xD
B\xF9\xDB\xB3\xDB\x9F\xAF/\xA4i8s\xA4\xEB\xF7\xA7W\x97gR\xA3\xD9n\x7F\
+xD0\xCE\xD
A\xED\xF3\xDBs\xE9_\xAFn__IJK\x96nB\xDF\36\x85\xED\xF6\xC5\x9B\x86\xD4
+\x98\x86\x
E1|\xD0n?>>\xB6\36\xB5\x96\xE7O\xDA\xB7\xEF\xDA\x9F(\27\x85\x92\xC5_\x
+9B\1\xA3iY
\xA1\xD58\xFE\xE69\xCB\xE4\xD3\xCCq\x83#\16\3\xA5\xDF\xEFGt\x8Do$\xF6\
+xC7\x92\16
\xBC\xC9*\xB57'\xEE\xC47\xE7\xD3\xB9\xEF\x85\xDE\xC8s\30a0\x9A\x92\x99
+\xD9\xDE\x
A6\32\17\xB7\xF2\30\x9B#2\xF4\xBC\xFB\xD6\xC8\x9B\xB5UY\xEE\xB5\xC7C\x
+C8\xE8\x98
\xD1<\x9F\22\xD3:\x8E\xC9\xE1\xE7\xDF\x9AM\xE9\xF4\xFD\xE5\xD5\xB9ty>\
+x90TU\x97\
xA5f3\xF5\xFE\xD5\xDB\x9B\xDB\x81\xA4t\xD5\x96\xD2i\xC9-E\xD6Y\x9AU\xA
+2U\xE2`\xE
4\xDB\xF3P
\x97sr\xD4\10\xC9\xA7\xB0\xFD\xD1|0\xA3\xA7\x8D\xE3\xAD\xF4\17\xA6/\x8
+5\xF6\x8C\
xF8\xD7\xE6\x84\xDC\x84\xA6\37JG\x92K\36\xA5s3$\x87OZ\23\22\xDE\xC2\xE
+B\xC3'\xCF
\xD6\xB2\xB4#N\xBC\x9Cg$4%Z\7M\xF2\xDF\x85\xFDp\xD4\xF8W\xF3\xFDI\xF3\
+xCC\x9B\xC
D\xCD\xD0\36:\xA4!\x8D<7$nx\xD4\30M}oF\x8E\x94\x86\xD4\xDE(g\1\6\x97\2
+7G\27\xB3\
x85\3r^^t\5\\\xCE"\xA2\xE6-TI\x83&Yqa\xD5C{\xC03i45\xFD\x80\x84G\xEFo_
+4{\x8D\xF6
F\xF9\x9E\x87v\xE8\x90\xE3S{rNL\x876\xA8\xD4\x94n\xCC\7"-\xE6R\xE8I}\x
+F9\x9F\x7F
\x93Nm\x8B~\xFF`\xBB\xD2{wH\xCC\xD0\4y%J\20@\xEA\r\xDA\xE7\xED...
(+ 24753 more bytes not shown)
!! Home Page !!

[download]

Replies are listed 'Best First'.
Re: https login using WWW::Mechanize
by codeacrobat (Chaplain) on Aug 09, 2010 at 06:32 UTC
[reply]
[d/l]
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: https login using WWW::Mechanize
by Khen1950fx (Canon) on Aug 09, 2010 at 07:01 UTC
    You're trying to login to a https site, so you have to tell LWP that: 
    #!perl

use strict;
use warnings;
use LWP::UserAgent;
use WWW::Mechanize;

my $ua = LWP::UserAgent->new;
$ua->protocols_allowed(['https']);

    [download]
[reply]
[d/l]
      I think WWW::Mechanize already understands that from the URL provided. I was able to login to a https website (https://mail.yahoo.com) already with the code posted there. Please read the detailed description of the issues I'm facing and the measures I've taken to debug the issues in my posting. Thanks.
[reply]
        I think WWW::Mechanize already understands that from the URL provided.

        What makes you think that? What have you done to test that hypothesis?

        Perl 6 - links to (nearly) everything that is Perl 6.
[reply]
Re: https login using WWW::Mechanize
by Anonymous Monk on Aug 09, 2010 at 13:52 UTC 
    $ua->add_handler("request_send",  sub { shift->dump; return });
$ua->add_handler("response_done", sub { shift->dump; return });

    [download]
[reply]
[d/l]

      Hey

      Thanks... I can now see the cookies getting padded to the requests, but on the third request (where I'm redirected to the homepage)I just see

      Cookie2: $Version="1".

      In response to that request, the http response cookie has the field "loggedin=false" which was 'true' in all the prior transactions. I was even able to see my user ID & other personal info fetched from the server in the cookie fields in the first 2 trransactions. So I assume I was logged into the server before this transaction happened. Am I still logged in ?

      'Client-SSL-Warning: Peer certificate not verified' for the first 2 requests. Is that an issue ?

      The field 'Connection' is always 'close' in response to my requests where I set that field to 'keep-alive'.

      How do I remain logged in to continue with further transactions ?

      Please help.... thanks

[reply]
        forgot to mention, that there is very little data in the 'cookie' field in the above mentioned GET request & Cookie2: $Version="1". The cookies in the prior requests contain a lot of information. This request appends barely any information in the cookies.

        I've disabled Java settings in the firefox browser & in firebug I can see the web server updating cookies pertaining to login, user info.... frequently.

        How do I get this thing working ?? Any help please....

[reply]

Back to Seekers of Perl Wisdom