I am using WWW::Mechanize to scrape from https://blah.foo.www.domain.com/main/index.html, which is a secured site that can only be obtained via Netegrity Siteminder-based authentication via https://blah.foo.www.domain.com/registration/login.html

So this is what normally happens if you go manually:

  1. you type in https://blah.foo.www.domain.com/main/index.html
  2. You are not detected to be logged in so you're automatically redirected to https://blah.foo.www.domain.com/registration/login.html
  3. You fill out an online form with your username and password
  4. You submit, which takes you to https://blah.foo.www.domain.com/login/login.fcc (Siteminder) which will check out your entry for authentication
  5. You are successfully logged in, so on you to go https://blah.foo.www.domain.com/main/index.html

I have a Perl script that should perform all steps 1 - 5, however it breaks at 5, never getting to the intended URL due to receiving a 404 Object Not Found error, furthermore, in my LWP Debug statements I get "LWP::UserAgent::request: Simple response: Not Found". Here is how I'm trying to do it using Perl: 

#!/strawberry/perl/bin/perl 


use strict; 
use warnings; 
use HTTP::Cookies;  
use WWW::Mechanize; 
use LWP::Debug qw(+); 


require "C:\\Documents and Settings\\me\\Desktop\\subs.pl"; 


if (my err = read_cfg("C:\\Documents and Settings\\me\\Desktop\ 
\file.cfg")) { 
          print(STDERR $err, "\n"); 
          exit(1); 



} 


my $mech = WWW::Mechanize->new(); 
$mech->agent_alias('Windows IE 6'); 
$mech->cookie_jar(HTTP::Cookies->new(autosave => 1)); 
$mech->add_header('UID' => $CFG::cfg{'USERNAME'}, 'cn' => 
$CFG::cfg{'CN'}); 

# FIRST PART: GO TO INDEX WHICH WILL REDIRECT YOU TO LOGIN AS YOU 
HAVEN'T YET 
my $response = $mech->get($CFG::cfg{'URL'}); 
die 'Error at ' , $CFG::cfg{'URL'}, "\n" $response->status_line, "\n 
Aborting" unless $response->is_success; 
print "\n\nSUCCESS data type: ", $response->content_type, "\n\n"; 


# SECOND PART: LOGIN FROM LOGIN PAGE WITH USERNAME AND PASSWORD 
PROVIDED 
$mech->submit_form( 
                                 form_number => 1, 
                                 fields => { 
                                                 USER => 
$CFG::cfg{'USERNAME'}, 
                                                 PASSWORD => 
$CFG::cfg{'PASSWORD'} 
                                 } 
); # WILL SUBMIT TO /registration/login_2 TO BUNDLE USERNAME/PASSWORD 
TO SEND TO SITEMINDER AGENT 


# THIRD PART: GO TO /registration/login_2 TO HANDLE LOGIN REQUEST WITH
+ 
FORM ENTRIES 
my $content = $mech->content; 
my $urlHeader = $CFG::cfg{'URL_HEADER'}; 
$content =!~ s[(href=['"]?)(/main)][$1$urlHeader$2]isg; 
$mech->update_html($content); 
$mech->submit(); # WILL SUBMIT TO SITEMINDER AGENT AND RETURN VERIFIED
+ 


$content = $mech->content;

[download]

---------------------------------------------------------------------------­---------------------------------------------------------------

However, it never successfully gets to Step 5), here is the output to hopefully explain why: 

C:\>perl "C:\\Documents and Settings\\me\\Desktop\\file.pl" 
LWP::UserAgent::new: () 
LWP::UserAgent::request: () 
HTTP::Cookies::add_cookie_header: Checking blah.foo.www.domain.com for
+ 
cookies 
HTTP::Cookies::add_cookie_header: Checking .foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking .www.domain.com for cookies
+ 
HTTP::Cookies::add_cookie_header: Checking www.domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .com for cookies 
LWP::UserAgent::send_request: GET https://blah.foo.www.domain.com/main
+/i 
ndex.html 
LWP::UserAgent::_need_proxy: Not proxied 
LWP::Protocol::http::request: () 
LWP::UserAgent::request: Simple response: Found 
LWP::UserAgent::request: () 
HTTP::Cookies::add_cookie_header: Checking blah.foo.www.domain.com for
+ 
cookies 
HTTP::Cookies::add_cookie_header: Checking .foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking .www.domain.com for cookies
+ 
HTTP::Cookies::add_cookie_header: Checking www.domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .com for cookies 
LWP::UserAgent::send_request: GET https://blah.foo.www.domain.com/regi
+stration/login_ 
1? 
TYPE=33554432&REALMOID=06-3fc2790b-96fd-0006-0000-1e2300001e23&GUID=&S
+MAUTH­REA 
SON=0&METHOD=GET&SMAGENTNAME=$SM$NKtSpmLX2VhxLeg5Fc91DK221%2bP2Wf 
%2bMwczu%2fbNNC 
LE%3d&TARGET=$SM$https%3a%2f%2fblah%2efoo%2ewww%2edomain%2ecom%2fmain 
%2f 
2f%2findex%2ehtml 
LWP::UserAgent::_need_proxy: Not proxied 
LWP::Protocol::http::request: () 
LWP::Protocol::collect: read 469 bytes 
LWP::UserAgent::request: Simple response: Found 
LWP::UserAgent::request: () 
HTTP::Cookies::add_cookie_header: Checking blah.foo.www.domain.com for
+ 
cookies 
HTTP::Cookies::add_cookie_header: Checking .foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking .www.domain.com for cookies
+ 
HTTP::Cookies::add_cookie_header: Checking www.domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .com for cookies 
LWP::UserAgent::send_request: GET https://blah.foo.www.domain.com/regi
+stration/login_ 
1/? 
TYPE=33554432&REALMOID=06-3fc2790b-96fd-0006-0000-1e2300001e23&GUID=&S
+MAUTH­RE 
ASON=0&METHOD=GET&SMAGENTNAME=$SM$NKtSpmLX2VhxLeg5Fc91DK221%2bP2Wf 
%2bMwczu%2fbNN 
CLE%3d&TARGET=$SM$https%3a%2f%2frup2%2edev1%2eprime%2eirs%2ecom 
%2fsemail%2fviews 
%2f%2findex%2ehtml 
LWP::UserAgent::_need_proxy: Not proxied 
LWP::Protocol::http::request: () 
LWP::Protocol::collect: read 4096 bytes 
LWP::Protocol::collect: read 284 bytes 
LWP::Protocol::collect: read 4096 bytes 
LWP::Protocol::collect: read 619 bytes 
HTTP::Cookies::extract_cookies: Set cookie JSESSIONID => 
0000CNUANTLZENJHYTKEG3S 
USXQ:10i31rv29 
LWP::UserAgent::request: Simple response: OK 


SUCCESS data type: text/htmlcharset=ISO8859-1 


LWP::UserAgent::request: () 
HTTP::Cookies::add_cookie_header: Checking blah.foo.www.domain.com for
+ 
cookies 
HTTP::Cookies::add_cookie_header: - checking cookie path=/ 
HTTP::Cookies::add_cookie_header:  - checking cookie 
JSESSIONID=0000CNUANTLZENJH 
YTKEG3SUSXQ:10i31rv29 
HTTP::Cookies::add_cookie_header:    it's a match 
HTTP::Cookies::add_cookie_header: Checking .foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking .www.domain.com for cookies
+ 
HTTP::Cookies::add_cookie_header: Checking www.domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .com for cookies 
LWP::UserAgent::send_request: POST https://blah.foo.www.domain.com/reg
+istration/login 
_2/0,,,00.html 
LWP::UserAgent::_need_proxy: Not proxied 
LWP::Protocol::http::request: () 
LWP::Protocol::collect: read 815 bytes 
LWP::UserAgent::request: Simple response: OK 
LWP::UserAgent::request: () 
HTTP::Cookies::add_cookie_header: Checking blah.foo.www.domain.com for
+ 
cookies 
HTTP::Cookies::add_cookie_header: - checking cookie path=/ 
HTTP::Cookies::add_cookie_header:  - checking cookie 
JSESSIONID=0000CNUANTLZENJH 
YTKEG3SUSXQ:10i31rv29 
HTTP::Cookies::add_cookie_header:    it's a match 
HTTP::Cookies::add_cookie_header: Checking .foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking .www.domain.com for cookies
+ 
HTTP::Cookies::add_cookie_header: Checking www.domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .com for cookies 
LWP::UserAgent::send_request: POST https://blah.foo.www.domain.com/log
+in/login.f 
cc 
LWP::UserAgent::_need_proxy: Not proxied 
LWP::Protocol::http::request: () 
HTTP::Cookies::extract_cookies: Set cookie FORMCRED => +Jy1O0pLR06/ 
i8PvjzNJjBkjQ 
gIms8tauo/Va9iPbL7dVx2DsUD2UTEg1ebOw 
+yVzRzlV3t7ziD8EFjzHX1WYBO50h2gCkRGgk2Z1BVCF 
FJ97ixaop+sW3F39bQWPGpanf6nDrJYkzY= 
LWP::UserAgent::request: Simple response: Found 
LWP::UserAgent::request: () 
HTTP::Cookies::add_cookie_header: Checking blah.foo.www.domain.com for
+ 
cookies 
HTTP::Cookies::add_cookie_header: - checking cookie path=/ 
HTTP::Cookies::add_cookie_header:  - checking cookie 
JSESSIONID=0000CNUANTLZENJH 
YTKEG3SUSXQ:10i31rv29 
HTTP::Cookies::add_cookie_header:    it's a match 
HTTP::Cookies::add_cookie_header: Checking .foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: - checking cookie path=/ 
HTTP::Cookies::add_cookie_header:  - checking cookie FORMCRED= 
+Jy1O0pLR06/i8Pvjz 
NJjBkjQgIms8tauo/Va9iPbL7dVx2DsUD2UTEg1ebOw 
+yVzRzlV3t7ziD8EFjzHX1WYBO50h2gCkRGgk 
2Z1BVCFFJ97ixaop+sW3F39bQWPGpanf6nDrJYkzY= 
HTTP::Cookies::add_cookie_header:    it's a match 
HTTP::Cookies::add_cookie_header: Checking foo.www.domain.com for 
cookies 
HTTP::Cookies::add_cookie_header: Checking .www.domain.com for cookies
+ 
HTTP::Cookies::add_cookie_header: Checking www.domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking domain.com for cookies 
HTTP::Cookies::add_cookie_header: Checking .com for cookies 
LWP::UserAgent::send_request: GET https://blah.foo.www.domain.com/main
+/i 
ndex.html 
LWP::UserAgent::_need_proxy: Not proxied 
LWP::Protocol::http::request: () 
LWP::Protocol::collect: read 42 bytes 
HTTP::Cookies::extract_cookies: Set cookie FORMCRED => 
HTTP::Cookies::extract_cookies: Set cookie smretry_cookie => true 
HTTP::Cookies::extract_cookies: Set cookie SMSESSION => 
NffP0fewmFwhTPvRVx/EXE1H 
5236J2xl/olTx6HGAv5Hs3pVASSK6Px4J/9kEX5VuYN/f 
+XgOIDojsKNrCqFZZK4vPxFtWBWIuifF/n2 
VWi0h02TP0TvDPhSQ63uzau1cHPygGFSBSA3qLl+g0S3s/llOT27KR6/ 
Vz35ddJWxiNHkU4b6f7G6tll 
/dKJjH7STmJ5stZNWIglZu7IAgRcnn5S7txqNynNkn9XevKI60q/ 
yMqTf1oIb5ZrMHRQbXpB3SEkBFyf 
COC9AR7/e2+U3qyMSAJ6p6I0iKNB/ 
GHzwWztJ6pEc1OJNLPE78BV9rXcom0FjWjOopWq3rrGormWKZPa 
tVY2K3+GeUfn/5ygqlTzjUycF+8c3v6J/PPGCZ+3bFQvIc8vxtXomCtkTQ4cSnUk/ 
mR2+MndMD5hK6Wc 
SxdaktyUEIF7d621AEvtxXFYVI98cb5a25l6jLWaVrMHN76wPns 
+DkitKgwswXyrgCNbGAFEjrgzBMbq 
qeFfzDOJXVUvsTuVXPKD2LJYcN4HwaoK1h3Nw+4NUhHGcz9tvisv20zGMMKIkyIMl/ 
LXVwe8vQ0MCme4 
0WFcIzkToDXZsSe7E7Mhk7D9t6u/SE2oeFheS07qyFW217MGTsOgbnN80Mv0N/ 
g6TsKxTOROB3QTuo43 
rXdOmH9zlmxtwP4uJ13KvN8JDtEdj5MK/ 
kN8ePQtuMK240f3vfcYqzI33nXnfhI2N6hkqo6yyqGjW0dO 
RQAMYn5/ 
qsI0yEbFf2h1oVECf8SMYF2XKEz32uMH9vwPC14FwGJPaVeOgB9QMrFgVFDyzf2xhrODtx
+cI 
Py+oMGxROADxOo1ibIrBEwBKlJlg7moahW0OnZRSF16n8R2yQu6ctuVy2xSGAErXEbWQA 
+YeeI+iaavK 
GaHyqNaYxv5M95SXg9GPj5H3oD2z1hyouU9JvUTsTGBdM8w2QwVCXcFmarP0Hb9s66YjAJ
+iFPCv­NztK8 
mwATiJvARlJV76TfVzu86BHD4+sPIWlyoehHDUSoPSaSlU/ 
SwWV05gFqgx8rNg26zmAchudkqZXFMLx8 
JdK80rs8x4JaBiIhNzfXyYzKo/qfm2T4X7D3E4smxdqWF9fzTaXd9ke5Wp/ 
iCDq9HfEbr0PTBXDpLl8u 
YdISKfD9alvHIop8d3JDucRMswbtx3K0Mgx+rQAvKMscAlvHVK/ 
TzZX6QBRUdvpzl1oWtO3bn3HNjdSv 
AFVK7dZCTZVcZw== 
HTTP::Cookies::extract_cookies: Set cookie smretry_cookie => true 
LWP::UserAgent::request: Simple response: Not Found

[download]

In reply to LWP::UserAgent::request: Simple response: Not Found by ppowell

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.