http://qs1969.pair.com?node_id=493084

cdherold has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I am finally getting a handle on WWW::Mechanize.

Now I want to go into my brokerage account and do some screen scraping.

But before I start testing my ability to get into my account by sending my USER/PASS out over the web using WWW::Mechanize, I thought it might be good to encrypt them.

I know little about encryption, save that it exists and that I should probably most definitely use it in this case.

The code that I am currently using is ...

$mech->submit_form( form_name => nameOFform, fields => { user => 'MyName', pass => 'MyPass', } );

I am curious what modifications I should make to encrypt these values as they course through the ether.

I ask for the monks' direction in further educating myself in this area ... and at the same time not setting myself up to see my brokerage account looted.

Most Humbly,

Chris Herold

Replies are listed 'Best First'.
Re: Encrypting User/Pass sent by WWW::Mechanize
by CountZero (Bishop) on Sep 19, 2005 at 06:33 UTC
    The only way you can protect your username and password from getting unencrypted out of your box into the wild internet is if you use a secure connection, such as https.

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

      Not quite true. If the website is using Digest authentication, then your password is never transmitted in the clear. However most websites won't accept that, so you have to send your password in the clear.

      If this distresses people, it is no worse than typing name/password into a normal web form. (Perhaps this distresses you even more...)

        Well, I said "such as https" and I count Digest authentication as some form of secure connection (at least as far as username/password is concerned and that was the scope of the question). I should have made that more clear perhaps.

        CountZero

        "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

      Does it count if the webpage I am logging onto is https? I am guessing not, but thought I should ask just to make sure.

      If not, How hard would it be for me to set it up?

      (A) It would be very hard.
      (B) It would be pretty hard, but go here for help.
      (C) It's so easy, let me show you how.

        I suppose there are two aspects to keeping this username and password combination secure:

        • Firstly, the local storage of the script and the username and password it contains.
        • Second, the transmission of these details across the wider internet.

        For the first item, I'd say it would be difficult to securely store the password in the script. Even if you were to use some sort of encryption (and I'm no expert on this) then by virtue of running the script it would have to automatically decrypt the password for use. Therefore you'd need to have some other password or code phrase to restrict access to it, and then you might as well not store your original password in the script and just type it in every time.

        The alternative option is that you store the script + plain text username and password on one of those USB flash memory sticks that do encryption. They either force you to enter a password or some biometric data (fingerprint) and then they appear as a regular drive on your system. You can then just run the script from the removable flash drive.

        Getting back to Perl for the second point, you'd need to ensure that the web site you're talking to uses HTTPS (they really should do) and then just use the secure URL with WWW::Mechanize and you'll know that the details are encrypted across the internet.

        You will need the Crypt::SSLeay and relevant libraries (OpenSSL or the DLLs for Windows - the latter are installed if you use PPM to get Crypt::SSLeay) to use WWW::Mechanize with SSL sites though.

      Should I be distressed? I thought I should be, but if it is the same as entering the data in a web browser as I do every day, then I feel less distressed ... I think.
Re: Encrypting User/Pass sent by WWW::Mechanize
by tomazos (Deacon) on Sep 19, 2005 at 13:15 UTC
    It sounds like the web page you are logging onto is using HTTPS (a secure encrypted connection).

    You can confirm this by looking at the first part of the Address bar in your web browser when you log into your brokerage account. If it says "https://" at the front, you are using HTTPS. If it says "http://" at the front, you are using an unencrypted connection.

    If it does say https, try substituting http in the Address bar of your web browser, and login again. If it doesn't work this means your server only allows encrypted login.

    Now check whether your username and pass are submitted via a GET or POST. You can do this by logging into your brokerage account and then look at the first Address after you login. If it contains your username somewhere in it, than it is GETing, if it doesn't than you are POSTing.

    Finally if you are POSTing and your server requires HTTPS, and your script in its current form works - than your username and password are already being encrypted by HTTPS and you do not need to do it a second time.

    In any case, chances are no-one is going to sniff your password. If a professional wanted to do it - they can always hack the DNS system to setup a fake proxy between you and the server. Several people in Switzerland have had their online banking hacked and money stolen by some Russians just recently using this technique. The banks used timed-sessions over HTTPS and had snail-mailed user IDs and passwords, but they still got through. Feel safe? :)

    -Andrew.


    Andrew Tomazos  |  andrew@tomazos.com  |  www.tomazos.com
Re: Encrypting User/Pass sent by WWW::Mechanize
by johnnywang (Priest) on Sep 19, 2005 at 19:49 UTC
    Your bank site must be using https, if not, switch banks, seriously. One common practice nowadays is that the bank homepages are usually not secure (e.g., http://www.bankofamerica.com), but the login forms are secure (just check the source for bankofamerica page). Of course, this is not really secure since someone can change the homepage, hence the form. I think there was a recent slashdot discussion on this practice. Anyhow, if you directly submit to the secure login form using https, you're safe. There is no need to change your code, just add "use Crypt::SSLeay;" to your script. You can install it (if on windows) by: ppm install http://theoryx5.uwinnipeg.ca/ppms/Crypt-SSLeay.ppd