Intaglio has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I'm writing a program for my own personal use that will grab stuff from various webpages and display just what I want so I can set it as my homepage and get my stock information, latest sports news, headlines, etc. all in one place everytime I startup IE. For this reason I'm trying to get this program to login to my excite.com page and grab that info from it. This is what I have so far: 

#!/usr/local/bin/perl -w

use HTTP::Request::Common qw( POST );
use LWP::UserAgent;

print "Content-type:text/html\n\n";

my $ua = new LWP::UserAgent;
my $req = POST 

'http://reg.excite.com/mps/login?a=a&pname=pfp&brand=excite&targeturl=
+http://www.excite.com/&ml_id=link:xcit:1008:i:',
     [ acctname => 'Intaglio', passwd => '******' ];
my $res = $ua->request($req)->content;

if ($res->is_success) {
     print $res->content;
} else {
     print $res->error_as_HTML;
}

[download]

The only problem is that it displays just a blank white page. Can anyone send a few helpful hints my way? Is it a problem that the page I'm sending the post information to is http://reg.excite.com/mps/login?a=a&pname=pfp&brand=excite&targeturl=http://www.excite.com/&ml_id=link:xcit:1008:i: or would that make a difference so long as its the correct page you can login to? There's also quite a bit of javascript password encryption on the login page, could this be a hindrance of my script?

Is what I want to do even possible?

Any help would be appreciated,

-- Intaglio

Replies are listed 'Best First'.
Re: LWP/login question
by chromatic (Archbishop) on Jan 11, 2001 at 10:04 UTC
    You're mixing query-string information with a POST request. People with better attention spans than I have read the relevant RFCs lately and aren't sure if you can do that.

    Depending on how well the Excite site handles CGI parameters, you may have to move completely to POST or GET.

[reply]
Re: LWP/login question
by zzspectrez (Hermit) on Jan 11, 2001 at 11:32 UTC

    There are a few problems... First off, if you examine the html source you would notice that before the form data is posted, a javascript routine is executed. This routine calls other javascript functions which appear to encrypt you password and username using MD5 digest. And also fill in some other hidden values. Im not familiar with MD5, or javascript so dont know how to solve your delema. But obviously you need to encrypt the data before it is sent.

    From the html source 
    <!--
// (c) Excite Inc., 1999
// ALL RIGHTS RESERVED
// THE JAVASCRIPT IMPLEMENTATION OF THE MD5 FUNCTION WAS DERIVED IN PA
+RT
// BY EXCITE FROM THE RSA DATA SECURITY, INC. MD5 MESSAGE DIGEST ALGOR
+ITHM
// AND IS USED BY PERMISSION.
de>

    [download]

    Im interested to see if any other monks could figure ot and show us how to use Digest::Perl::MD5 or Digest::MD5 and lwp to automate login.

    zzSPECTREz

[reply]
[d/l]
      You might be able to get around this by saving the form to your own computer, and changing the <form action=""> tag to point to your own cgi-script which will echo the parameters as they are being passed by the form to excite so you know what they are being changed to by the javascript.

      Might do the trick, just an idea.

[reply]

Back to Seekers of Perl Wisdom