in reply to Confused about LWP, Mechanize and UserAgent

You can't do all this with LWP::Simple. LWP::Simple is only intented to be used for simple (heh) requests, but even though it does use LWP::UserAgent behind the scenes there's no access to the more advanced UserAgent features you'll need.

You really should take a look at WWW::Mechanize. Mechanize is a subclass of UserAgent that's designed to make just this kind of "walking a site" tasks easy. It also handles cookies by default, so for most sites you shouldn't have any problems handling the session/login information (mechanize should just do the right thing).

Example:

my $m = WWW::Mechanize->new(); # point to the url of the login form $m->get("http://www.mydomain.com/login_form.html"); # submit form with data specified $m->submit_form( form_name => "login_form", { username => "xxxxxxx", password => "yyyyyyy", } ); # now you are logged in. # do actions that you need to be logged in for $m->get("http://www.mydomain.com/program.cgi?something"); # or $m->follow_link( text => "Click here to perform some action");

Once you're familiar with WWW::Mechanize, it's a lot easier to automate web applications with it than with LWP::UserAgent, and you can still use (almost) all of the tricks you can do with UserAgent too.