PerlNuBe has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks - I have been trying to setup a crawler to check all my info on the web. I want it to get it daily. So far, it works like it's suppose to. Except once license site which has a page loading from a query from their database. After much digging I see SOAP everywhere. I am posting my code below. Any ideas?

Thanks in advance.

use WWW::Mechanize; $browser = WWW::Mechanize->new(autocheck=>1); $URL = "my.license site.gov"; $browser->get("$URL",":content_file" => "sample1.html"); $browser->timeout(10); $browser->agent('User-Agent=Mozilla/5.0 (Windows; U; Windows N +T 5.1; en-US; rv:1.8.1.4) Gecko/20070515 Firefox/2.0.0.4'); # $browser->form_number(3); # $browser->field("P_CITY"=>"Merced"); # $browser->submit(); $browser->getFrame('name:',''); $results = $browser->content; print "$browser"; open(OUT,">Results.html");print OUT $results;close OUT;

Replies are listed 'Best First'.
Re: Crawling a site with Soap and IFrames
by Anonymous Monk on Jul 29, 2010 at 06:57 UTC