Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

How can I download a web page which can be accessed only after entering password and login on the main page. I tried this the authentication_basic() method, but it does not work. Can anyone help me please? Thanks
  • Comment on Downloading a web page from a secure site

Replies are listed 'Best First'.
Re: Downloading a web page from a secure site
by davorg (Chancellor) on Oct 16, 2001 at 20:27 UTC

    authentication_basic works when the page is behind an Apache-style Basic Authentication scheme. These are the pages where you get a dialog box asking for your username and password. It sounds like the authentication on your page isn't the same ("...entering password and login on the main page"). You'll need to work out how the site handles authentication and write some LWP requests that mimic those methods.

    --
    <http://www.dave.org.uk>

    "The first rule of Perl club is you don't talk about Perl club."

Re: Downloading a web page from a secure site
by cacharbe (Curate) on Oct 16, 2001 at 20:24 UTC
    It would be nice if we could see what you have tried, but look here for some LWP code that works for me. It posts user info to a website, opens a cookie jar, etc. Second time in 24 hours I've linked someone to this node...

    C-.

lwpcook is your friend
by Fletch (Bishop) on Oct 16, 2001 at 21:29 UTC
    $ perldoc lwpcook ... ACCESS TO PROTECTED DOCUMENTS Documents protected by basic authorization can easily be accessed like this: ...
Re: Downloading a web page from a secure site
by earthboundmisfit (Chaplain) on Oct 16, 2001 at 20:03 UTC
    Try posting some code between <CODE> tags and
    remember, Super Search is your friend :)
Re: Downloading a web page from a secure site
by jplindstrom (Monsignor) on Oct 17, 2001 at 16:39 UTC
    Look at the login page HTML and figure out what form elements are used for the login. Mimic this behaviour with a POST or GET using LWP. You may have to fake HTTP_REFERER.

    Step two, they probably use cookies for keeping the user session ID, so take a look in the LWP docs for how to attach a cookie jar to your user agent.


    /J