LWP posting

PerlSufi has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I have been at this for a little while, and I am stuck on one part. I am trying to crawl a page to access a csv file. Here is my script (or part of it):

my $ua = LWP::UserAgent->new(ssl_opts => { verify_hostname => 1 });
$ua->agent('Mozilla/5.0');
$ua->cookie_jar({});
my $response = $ua->get('firstpage.com');
my $response2 = $ua->get('secondpage.com');

(etc)
###This section is where I am having trouble:

my $post = $ua->post('https://www.foo.com/downloadAcsReportSearch.html
+', "Search");
[download]

The problem I am having right now is that I am not clicking the "Search" button at the last post. Firebug shows it has the value 'Search'. I think I need help understanding how to do this. I also have to click one last button on the page proceeding that, but I'll cross that bridge when I get there. Any insight is greatly appreciated.Thanks!
PS-I should add that I cannot use a module that utilizes a browser or javascript plugin.

Comment on LWP posting Download Code

Replies are listed 'Best First'.
Re: LWP posting by aitap (Curate) on May 14, 2013 at 19:14 UTC
Try WWW::Mechanize: it's easier to use when you need to emulate a browser and sequentally fetch pages from different forms, saving cookies and other session-related parameters between calls. It also contains a handy utility calles mech-dump which will help you to fill the form properly.	[reply]
Re: LWP posting by SnackySmorez (Initiate) on May 14, 2013 at 19:28 UTC
Hi PerlSufi, try encoding the url. I seem to remember having to encode urls for some sites before submitting it. Also as the other poster replied, try Mechanize. If you have a choice use Mechanize.	[reply]
Re^2: LWP posting by PerlSufi (Friar) on May 14, 2013 at 19:59 UTC
changing the crawler to use WWW::Mechanize for my sites gave me this error: `Error GETing https://prodp1.usps.com/adminweb/view.htm?requestPage=P1D +ASHBOARD: Can't connect to prodp1.usps.com:443 (certificate verify failed) at ac +s_get.pl l ine 44.` [download] I think I may need to use LWP instead. Maybe with HTML::Form? And actually, I should mention that I use mechanize to login. Just haven't had much success navigating with it because of js stuff	[reply] [d/l]
Re^3: LWP posting by Anonymous Monk on May 14, 2013 at 20:20 UTC
... certificate verify failed ... That's like a FAQ item having to do with HTTPS "certificate verify failed" error while trying to connect to https:// site using use WWW::Mechanize, "certificate verify failed" difference between Perl 5.14 and 5.10 Also, mechanize is lwp	[reply]
Re^4: LWP posting by PerlSufi (Friar) on May 14, 2013 at 20:40 UTC
Re^2: LWP posting by PerlSufi (Friar) on May 14, 2013 at 21:27 UTC
Hi, Thanks for your insight. Actually, the header does say it is form encoded.. but how do I do that?	[reply]
Re^3: LWP posting by SnackySmorez (Initiate) on May 15, 2013 at 22:23 UTC
You can try forming your own url string and using url::encode. http://search.cpan.org/~chansen/URL-Encode-0.01/lib/URL/Encode.pod. Also worth checking out is uri::escape for %-encode and %-decode unsafe characters http://search.cpan.org/~gaas/URI-1.60/URI/Escape.pm	[reply]
Re: LWP posting by PerlSufi (Friar) on May 15, 2013 at 00:35 UTC
I got throught that part by adding `use IO::Socket::SSL qw(); my $mech = WWW::Mechanize->new(ssl_opts => { SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE, verify_hostname => 0,});` [download] So I'm not there yet but I'm further than I was.. Thanks!	[reply] [d/l]