Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,...

Please I ask for you patience and wisdom...

I have this script; 
require LWP::UserAgent;
 
 my $ua = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 
 my $response = $ua->get('http://toolsview/bcp_tool/');
 
 if ($response->is_success) {
     print $response->content;  # or whatever
 }
 else {
     die $response->status_line;
 }

[download]
the output I get is; 
<html>
<head>
<title>BCP Administrator Console</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1
+">
</head>
<frameset rows="90,*" frameborder="NO" border="0" framespacing="0" col
+s="*">
  <frame name="top" scrolling="NO" noresize src="top.asp" frameborder=
+"NO" >
  <frame name="main" src="logon.asp">
</frameset>
<noframes>
<body bgcolor="#FFFFFF" text="#000000">
</body>
</noframes>
</html>

[download]
Which is a web page that requests users to login (i.e. to insert a user ID and a password then click on the 'Login' button).

My question is, can I automate the login process using Perl and how would I be able to do this? - This is only a prelude to more complicated things to come with LWP - and I will continue to read LWP documentation. However the answer to my enquiry will be a good spring board to launch me into Web programming.

Thanks

Replies are listed 'Best First'.
Re: My first LWP script
by Joost (Canon) on Jun 15, 2004 at 16:13 UTC
    LWP is very useful, but for your purpose you might be better off using WWW::Mechanize.

    It abstracts away a lot of parsing and HTTP stuff, so you can do this (from the docs): 

        use WWW::Mechanize;
    my $mech = WWW::Mechanize->new();

    $mech->get( $url );

    $mech->follow_link( n => 3 );
    $mech->follow_link( text_regex => qr/download this/i );
    $mech->follow_link( url => 'http://host.com/index.html' );

    $mech->submit_form(
        form_number => 3,
        fields      => {
            username    => 'yourname',
            password    => 'dummy',
        }
    );

    $mech->submit_form(
        form_name => 'search',
        fields    => { query  => 'pot of gold', },
        button    => 'Search Now'
    );

    [download]
    "What should it profit a man, if he should win a flame war, yet lose his cool?"
[reply]
[d/l]
Re: My first LWP script
by atcroft (Abbot) on Jun 15, 2004 at 16:19 UTC

    Look at lwpcook-there are examples with basic authentication from the webserver, or where POST is being used and the authentication is being provided by the pages themselves. I suspect these may at least help you get started. Alternately, it is my understanding that WWW::Mechanize might also be useful, with a recent article appearing on http://www.perl.com/ regarding HTTP::Recorder to generate WWW::Mechanize scripts.

    Hope that helps....

[reply]
      I'll second that emotion. HTTP::Recorder is a wonderfully handy interface for WWW::Mechanize scripts. Here's a link to the article itself:

      http://www.perl.com/pub/a/2004/06/04/recorder.html

      As the article points out, you may have some difficulty if you are using SSL to access this page. In that circumstance, you'll have to go through the HTTP::Recorder control panel, not just through the proxy.

      I've used this set-up several times, and it's fast, and easy.

[reply]
        This is great and I'll be trying it tonight. I just wish I would be able to have a macro function from within Firefox. Just select "record actions". Run through the actions on the website and save it.
[reply]
        WOW.
        I am consistently surprised with just how many cool things I'm exposed to on this site. The link to the perl.com article is something that will scratch a particular itch I've had for a while.
        Thank you.


        Very funny Scotty... Now PLEASE beam down my PANTS!
[reply]
Re: My first LWP script
by Qiang (Friar) on Jun 15, 2004 at 19:35 UTC
    LWP is perfect use for web automation. usually for web login, you need to take care the referer/browser user agent/cookie as the web application may check if this is a valid request ( shallow but does work sometimes ). use ethereal to sniff the packet if you can visit from brower but can't with LWP.

    for more info on how to do that check out %perldoc lwpcookbook and this article on perl.com by Sean M. Burke. I found it's very useful for people who are new to LWP. if you want to get detail on LWP and all the html processing, check his book "Perl & LWP".

[reply]
      If you can log in with the lynx text browser, then you can turn on its trace mode and get a trace file (~/Lynx.trace) that shows exactly what's going on. 
      lynx -trace http://foo.bar/

      [download]
      The trace file is a bit cluttered with debug info concerning how Lynx parses and renders HTML, but it is still pretty easy to follow.
[reply]
[d/l]
        forgot to mention.. alternative, if you use firefox from mozilla, there is the 'Live Http Header' plugin. it's simpler and easier to use.
[reply]
Re: My first LWP script
by Anonymous Monk on Jun 16, 2004 at 01:07 UTC

    You can also use HTTP::Request::Common to perform the automatic login process.

    use HTTP::Request::Common;
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$ua->request(POST 'http://url/loginscript', [ username => 'yourusernam
+e', password => 'yourpassword' ]);

    [download]

    - Ralph.

[reply]
[d/l]

Back to Seekers of Perl Wisdom