Monks,

I return from my quest with success!! It took me a couple of days and many ponderings on Monks' responses to my previous postings, but I now have code that will enter into my selected user/pass protected https webpage and cruise around internally to do almost whatever I want or need. I thought I would post and share as there were no definitive answers during my quest on how to get this done.

Comments for improvement are welcome.

#!/usr/bin/perl 

use strict;

use Crypt::SSLeay;

use LWP::UserAgent;
use WWW::Mechanize;
use HTTP::Cookies;
use HTTP::Request;

##The first two lines of code are used because
##I was sending my results to a web browser.

use CGI::Carp qw/ fatalsToBrowser /;

print "Content-type: text/html\n\n";

my $user = 'MyUser';
my $pass = 'MyPass';
my $base_address = 'https://wwws.stocksRus.com';

my $output = '';
my $dv_data = '';
my $webpage = '';

my $url = 'https://wwws.stocksRus.com/cgi-bin/LogIn';

my $agent = WWW::Mechanize->new( autocheck => 1); 

# Set up cookie jar
$agent->cookie_jar(HTTP::Cookies->new);  

##Go through login to get appropriate cookies 
##so that I can then move onto
##the following pages with my cookie jar properly loaded. 
    
$agent->get($url);
    die $agent->response->status_line unless $agent->success;

#In order to login properly it was necessary to send
#a hidden input called DV_DATA along with user/pass that
#appeared to be time based and was assigned on entry to
#the main login page. I extracted this value and 
#assigned it as one of the inputs for the form.
#This is a critical component of the login.
    
$output = $agent->content;

for ($output =~ /name=\"DV_DATA\" type=\"hidden\" VALUE=\"(.*?)\">/smi
+){
    
    $dv_data = $1;
    
}
    
$agent->form_name( 'LoginFormName' );

$agent->set_fields(
    USERID => $user,
    PASSWORD => $pass,
    DV_DATA => $dv_data
);    


$agent -> submit();   

###END Login 
##I now have the appropriate cookies and may continue.

#On the main page there is some javascript that codes for
#the pull down menus. I could not access these using
#WWW::Mechanize and one of them had the link I needed. 
#To solve, I logged into the webpage in a normal browser
#and found the url of the page that I wanted and plugged
#it in below. 
##Accessing this page directly does not work as there
##is some Authorization procedure that requires going
##through login first.

$agent->get('https://wwws.stocksRus.com/cgi-bin/Quotes');
    die $agent->response->status_line unless $agent->success;

#This page has a form that holds a symbol that I can 
#submit on to get data on that symbol.
    
$agent -> field    ('ticker','AMGN');

$agent -> submit();

#At first the webpage wouldn't come up in my web browser
#(which is where the output from this run was sent), 
#because I didn't have the base URL. All the addresses
#were similar to "/cgi-bin/menu/equity".
#I substitute the base URL in below.
 
$webpage =  $agent -> content;

$webpage =~ s/\"\//\"$base_address\//smig;
$webpage =~ s/\'\//\"$base_address\//smig;

#Prints out my webpage of interest.
print $webpage;        
    
exit;

[download]
Chris Herold

In reply to WWW::Mechanize to Access HTTPS with Cookies by cdherold

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.