in reply to Crawler Help

Welcome at perlmonks.

Note that perlmonks is not a code writing service. Show us your code, wrapped in <code> tags, and show us what is wrong with that code (error messages, unexpected output). Don't ask us how to write malicious code, we won't help you with that, for obvious reasons.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Replies are listed 'Best First'.
Re^2: Crawler Help
by Anonymous Monk on Jun 30, 2010 at 05:07 UTC

    Hi, When I try to fetch the page using any browser I get the result whereas when I try to fetch it using code I don't see any content.

    #!usr/bin/perl -w use strict; use WWW::Mechanize; my $ua = WWW::Mechanize->new; my $base_url = 'http://search.justdial.com/srch/search_cwa.php?srch=tr +ue&city=Chennai result=&city=&searchtype=Multicuisine+Restaurants&res +ultType=&area=&new_area=&Stype=category&choices=&intermid=true&docId= +&smsAlert=&emailAlert=&NameAlert=&sinfo=&smsDetail=&img=&img_check=&s +eeAlso=&sms_visible=&sms_button=&Lowcat=&lowest=&return_page=&magic_m +obile=&magic_email=&magic_sinfo=&magic_Detail=&magic_seeAlso=&magic_n +ame=&magic_tel=&login_name=&login_pass=&login_name_review=&login_pass +_review=&img_forget=&screen_name=&login_name1=&login_pass1=&comp=&cid +=&review_div=&ADVSEARCH=&city=Chennai&txtname=&txtadd=&Street=&area=& +Pin=&txt_company=&txtphone=&jdG=&search_type_string_company=&search_t +ype_string='; $ua->cookie_jar(HTTP::Cookies->new()); $ua->default_header('User-Agent' => 'Mozilla/5.0 (Windows; U; Windows +NT 6.1; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/3.0.195.3 +8 Safari/532.0'); open(DATABASE, ">just_dial.html") || die "$!\n"; DATABASE->autoflush(1); $ua->get($base_url); print DATABASE $ua->content();

      Does the site use Javascript? Have you tried visiting and operating the site without Javascript from a browser?