Perl for automated web browsing?

xgamma has asked for the wisdom of the Perl Monks concerning the following question:

Hi folks,

I wish to build a program that would login to a website, enter text into a field and submit, strip the data presented and paste it into a text file. Is this easily and intuitively feasible in Perl? If so, could you be so kind as to point me in the general direction to start my reading? My searches have so far been unsuccessful.

Thanks

Comment on Perl for automated web browsing?

Replies are listed 'Best First'.
Re: Perl for automated web browsing? by pc88mxer (Vicar) on Jun 16, 2008 at 18:32 UTC
Look up WWW::Mechanize. If the site doesn't require Javascript, it might work for you.	[reply]
Re: Perl for automated web browsing? by b10m (Vicar) on Jun 16, 2008 at 19:29 UTC
WWW::Mechanize would probably be the best (as suggested), but if the site does insist on Javascript, Mozilla::Mechanize or Win32::IE::Mechanize might help (note: I have never used any of those two; WWW::Mechanize suits me well). Update: ... and as for stripping the necessary data, please do have a look at Tatsuhiko's Web::Scraper. It might be a little tricky at first, but is a true gem which I learn to appreciate more and more everyday! -- b10m	[reply]
Re: Perl for automated web browsing? by mneuwirth (Initiate) on Jun 17, 2008 at 13:01 UTC
I like Win32::IEAutomation too, it is pretty straight forward and simple. I am pretty sure it can handle java also.	[reply]
Re^2: Perl for automated web browsing? by xgamma (Novice) on Jun 18, 2008 at 17:23 UTC
Unfortunately the site I'm using does use Javascript. For example, the login control is Javascript. So I guess Mechanize is out?	[reply]
Re^3: Perl for automated web browsing? by runrig (Abbot) on Jun 18, 2008 at 21:51 UTC
Even with Javascript, it's still just http GET's and POST's between the client and the server. Either read the Javascript and figure out what needs to be sent and/or use something like FireFox's Live HTTP Headers plugin to see what is being sent when you're navigating the site, and then send those requests yourself using WWW::Mech or LWP::UserAgent.	[reply]
Re: Perl for automated web browsing? by shekarkcb (Beadle) on Jun 17, 2008 at 07:18 UTC
Hi dude.... Surely .. You can look into perl module WWW::Mechanizedigg it where you have every methods and ways to scrape a web site. go for serach.cpan.org then in the search bar type above name u will get it surely. Don't forget also to read HTTP::Cookies shekar	[reply]


Syntactic Confectionery Delight
	PerlMonks