I'm confused as to how to access the web pages on a website that I need to parse. I have the username and password and I know I need to use the libraries above (useragent, mechanize, simple), but I'm seeing so many different ways of doing things and everything I'm trying is not working.

If the name of the site is www.mydomain.com and on this page there is a login screen (username, password) what do I need to add to my program so that it 'logs in'. My program grabs certain pages from the site and parses them. What I've been doing is passing the LWP::Simple library url's that look like this:

http://www.mydomain.com/program.cgi?var1=apples&var2=bananas

It's been working great and I'm able to get many of the pages on this site. However, like I mentioned above, some of the pages on this site are 'restricted' and I need to 'login' using my parsing program so that the 'get' command will work.

Can anyone give me a good, simple example of how to login so that I can use the LWP::Simple library on restricted pages? I just want to be able to use LWP::Simple without any access problems.

In reply to Confused about LWP, Mechanize and UserAgent by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.