LWP SSL TImeouts

amsclark has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm a law student trying to learn Perl. Hilarious, huh?

Anyway, I'm trying to write a script to automate looking up the status of court cases online at https://www.dccourts.gov/cco/maincase.jsf. But when I run my code, it just times out and I'm not sure why. The error I get is "error: 500 SSL read timeout: at ./minimaltestcase.pl line 34"

I thought maybe I had done my form pairs improperly, but it doesn't seem like that should be the cause of this problem since with other websites it seems like I can use the POST method even with BS values like 'foo' => 'bar'. I still get something back. If I try to POST to cnn.com or paypal.com, with this script, I do get HTML printed to the console.

Anyway, here's a stripped down version of the script. If anybody can point out an error or point me in a different direction, I would much appreciate it. I did try to use LWP::Debug, but it didn't output anything.

#!/usr/bin/perl -w
use warnings;
use strict;
use LWP::UserAgent;



my $url = 'https://www.dccourts.gov/cco/maincase.jsf';
#my $url = 'https://www.paypal.com/';

my $browser = LWP::UserAgent->new;

my $response = $browser->post($url, 
  [ 'appData%3Asearchform' => 'appData%3Asearchform',
  'appData%3Asearchform%3AsearchPanelCollapsedState' => 'false',
  'appData%3Asearchform%3Ajspsearchpage%3AlastName' => '',
  'appData%3Asearchform%3Ajspsearchpage%3AcompanyName' => '',
  'appData%3Asearchform%3Ajspsearchpage%3AfirstName' => '',
  'appData%3Asearchform%3Ajspsearchpage%3Aj_id_id14pc4' => '2011',
  'appData%3Asearchform%3Ajspsearchpage%3AselectCaseType' => 'CF2',
  'appData%3Asearchform%3Ajspsearchpage%3Aj_id_id18pc4' => '003817',
  'appData%3Asearchform%3Ajspsearchpage%3AnameAttributesPanelCollapsed
+State' => 'true',
  'appData%3Asearchform%3Ajspsearchpage%3AsubmitSearch.x' => '25',
  'appData%3Asearchform%3Ajspsearchpage%3AsubmitSearch.y' => '26',
  'javax.faces.ViewState' => 'j_id1%3Aj_id2',
  ]
);

#my $response = $browser->post($url, 
#  [ 'foo' => 'bar',
#  ]
#);

die "$url error: ", $response->status_line unless $response->is_succes
+s;

print $response->content;
[download]

Comment on LWP SSL TImeouts Download Code

Replies are listed 'Best First'.
Re: LWP SSL TImeouts by senthilkumarperl (Novice) on May 19, 2011 at 07:03 UTC
hi, hello use this code. It will work use strict; use LWP::UserAgent; use URI::URL; use HTTP::Cookies; my $ua = LWP::UserAgent->new; $ua->agent("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1 +6) Gecko/20080702 Firefox/2.0.0.16 (.NET CLR 3.5.30729)"); my $cookie_jar = HTTP::Cookies->new(file=>$0."_cookie.txt",autosave => + 1,); $ua->cookie_jar($cookie_jar); my $req=HTTP::Request->new(POST=>"https://www.dccourts.gov/cco/maincas +e.jsf"); $req->header("Content-Type"=> "application/x-www-form-urlencoded"); $req->header("Cookie"=>"JSESSIONID=9kpdNJ7dLQRc62Bsj3mTGsPNRnsvKyvL2DD +JtLPgLhXpD9nwGSWm!-462614910"); $req->content("appData%3Asearchform=appData%3Asearchform&appData%3Asea +rchform%3AsearchPanelCollapsedState=false&appData%3Asearchform%3Ajsps +earchpage%3AlastName=&appData%3Asearchform%3Ajspsearchpage%3AcompanyN +ame=&appData%3Asearchform%3Ajspsearchpage%3AfirstName=&appData%3Asear +chform%3Ajspsearchpage%3Aj_id_id14pc4=2011&appData%3Asearchform%3Ajsp +searchpage%3AselectCaseType=CF2&appData%3Asearchform%3Ajspsearchpage% +3Aj_id_id18pc4=003817&appData%3Asearchform%3Ajspsearchpage%3AnameAttr +ibutesPanelCollapsedState=true&appData%3Asearchform%3Ajspsearchpage%3 +AsubmitSearch.x=43&appData%3Asearchform%3Ajspsearchpage%3AsubmitSearc +h.y=12&javax.faces.ViewState=j_id1%3Aj_id3"); my $res=$ua->request($req); $cookie_jar->extract_cookies($res); $cookie_jar->save(); $cookie_jar->add_cookie_header($req); my $code=$res->code(); print "code:$code\n"; my $cont=$res->content(); open(postcontent,">>postcontent.html"); print postcontent $cont; [download] Regards, Senthil	[reply] [d/l]
Re^2: LWP SSL TImeouts by Mr. Muskrat (Canon) on May 19, 2011 at 18:56 UTC
"hello use this code. It will work" senthilkumarperl, amsclark made it clear that he's trying to learn Perl. So I suggest that you explain why it will work when the code in the OP does not.	[reply]
Re^3: LWP SSL TImeouts by Anonymous Monk on May 19, 2011 at 19:21 UTC
Hey, it's alright. I can learn pretty well from posts that say things like "use this, it will work" as well. From Senthil's code I started thinking about the necessity of allowing the server to set a cookie probably. It was a bump in the right direction I think.	[reply]
Re^4: LWP SSL TImeouts by amsclark (Initiate) on May 19, 2011 at 19:24 UTC
Re^2: LWP SSL TImeouts by Anonymous Monk on May 19, 2011 at 19:15 UTC
Senthil, Thanks so much for your response. I tried running your code and it did basically the same thing. It just sat there for a while and eventually outputted "code:500" and terminated. Here's what it outputted to the "postcontent.html" file: SSL read timeout: at /usr/local/share/perl/5.10.1/LWP/Protocol/http.pm + line 305 at /usr/local/lib/perl/5.10.1/Net/SSL.pm line 211 Net::SSL: +:die_with_error('Net::SSL=GLOB(0x2c0c6b0)', 'SSL read timeout') calle +d at /usr/local/lib/perl/5.10.1/Net/SSL.pm line 219 Net::SSL::__ANON_ +_('ALRM') called at /usr/local/lib/perl/5.10.1/Net/SSL.pm line 222 ev +al {...} called at /usr/local/lib/perl/5.10.1/Net/SSL.pm line 222 Net +::SSL::read('LWP::Protocol::https::Socket=GLOB(0x2c18990)', '', 1024, + 0) called at /usr/local/share/perl/5.10.1/LWP/Protocol/http.pm line +305 LWP::Protocol::http::request('LWP::Protocol::https=HASH(0x2a658f8 +)', 'HTTP::Request=HASH(0x29eaee8)', undef, undef, undef, 180) called + at /usr/local/share/perl/5.10.1/LWP/UserAgent.pm line 193 eval {...} + called at /usr/local/share/perl/5.10.1/LWP/UserAgent.pm line 192 LWP +::UserAgent::send_request('LWP::UserAgent=HASH(0x29c7ef8)', 'HTTP::Re +quest=HASH(0x29eaee8)', undef, undef) called at /usr/local/share/perl +/5.10.1/LWP/UserAgent.pm line 275 LWP::UserAgent::simple_request('LWP +::UserAgent=HASH(0x29c7ef8)', 'HTTP::Request=HASH(0x29eaee8)', undef, + undef) called at /usr/local/share/perl/5.10.1/LWP/UserAgent.pm line +283 LWP::UserAgent::request('LWP::UserAgent=HASH(0x29c7ef8)', 'HTTP:: +Request=HASH(0x29eaee8)') called at ./lookerupper2.pl line 16 at /usr +/local/lib/perl/5.10.1/Net/SSL.pm line 222 Net::SSL::read('LWP::Proto +col::https::Socket=GLOB(0x2c18990)', '', 1024, 0) called at /usr/loca +l/share/perl/5.10.1/LWP/Protocol/http.pm line 305 LWP::Protocol::http +::request('LWP::Protocol::https=HASH(0x2a658f8)', 'HTTP::Request=HASH +(0x29eaee8)', undef, undef, undef, 180) called at /usr/local/share/pe +rl/5.10.1/LWP/UserAgent.pm line 193 eval {...} called at /usr/local/s +hare/perl/5.10.1/LWP/UserAgent.pm line 192 LWP::UserAgent::send_reque +st('LWP::UserAgent=HASH(0x29c7ef8)', 'HTTP::Request=HASH(0x29eaee8)', + undef, undef) called at /usr/local/share/perl/5.10.1/LWP/UserAgent.p +m line 275 LWP::UserAgent::simple_request('LWP::UserAgent=HASH(0x29c7 +ef8)', 'HTTP::Request=HASH(0x29eaee8)', undef, undef) called at /usr/ +local/share/perl/5.10.1/LWP/UserAgent.pm line 283 LWP::UserAgent::req +uest('LWP::UserAgent=HASH(0x29c7ef8)', 'HTTP::Request=HASH(0x29eaee8) +') called at ./lookerupper2.pl line 16 [download] I used the Mozilla LiveHTTP Headers plugin to get a new JSESSIONID and replaced it in the code you provided but it still does the same thing: terminates with "code:500" and outputs the above into postcontent.html. I also tried passing the entire string for "Cookie" that I found in the HTTP Headers. That is, I tried a line that reads like this: `$req->header("Cookie"=>"__utma=189426622.1597458921.1305692566.1305781 +465.1305829478.4; __utmz=189426622.1305692566.1.1.utmcsr=google\|utmcc +n=(organic)\|utmcmd=organic\|utmctr=dc%20superior%20court");` [download] Same problem. Out of curiousity, I tried looking up a case in Firefox with javascript turned off and it worked fine. I also tried looking up a case in Lynx and it worked fine.	[reply] [d/l] [select]