Web Crawaling problem with JustDial

hemant.bhargava7 has asked for the wisdom of the Perl Monks concerning the following question:

Champs .. I just want to know that how to write a web crawler for justDial.com .. Actually i don't know how to pass values in those textboxes to next page .. i mean how i can automate that page .. I can automate eveything with Mechanize but Mechanize does not supports javascript .. So cant handle that also .. I am writing my code as below ..


!usr/bin/perl
use strict;

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use URI::Heuristic;
use HTTP::Cookies;

# Make an appropriate URL for you ..
my $rawURL = shift or die "Pass an URL\n";
my $url    = URI::Heuristic::uf_urlstr ($rawURL);
$| = 1;
printf "%s => \n", $url;

my $ua = LWP::UserAgent->new();
my $req = POST 'http://www.justdial.com';
                [ city => 'Bangalore', q => 'Bhargava' ];
my $content = $ua->request($req)->as_string;
print $content;
[download]

So if anybody can tell me even an inch of hint .. Then i'll be very thankful .. Cheers ..

Comment on Web Crawaling problem with JustDial Download Code

Replies are listed 'Best First'.
Re: Web Crawaling problem with JustDial by roboticus (Chancellor) on Apr 20, 2009 at 12:26 UTC
hemant.bhagrava7: Super search (the upper-right url on the screen) ought to give you far more than an inch of a hint... ...roboticus	[reply]
Re^2: Web Crawaling problem with JustDial by Anonymous Monk on Apr 20, 2009 at 12:39 UTC
Great tips in this thread How to handle javascript -- a foot(12")?	[reply]
Re^2: Web Crawaling problem with JustDial by hemant.bhargava7 (Initiate) on Apr 20, 2009 at 12:50 UTC
Upper right in URL .. Is it you are talking about .. See, What i wanted to do is:- i need to pass some params according to their names to this page (like 'q' => 'hemant') .. OK .. And then the current form will go to the next page and display all the results to the screen .. Ok .. If you know it .. Can you give me a small snippet for it .. Thanks in Advance ..	[reply]
Re^3: Web Crawaling problem with JustDial by Anonymous Monk on Apr 20, 2009 at 12:57 UTC
mechanize example search: www::mechanize google search	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Problem In Web Crawling .. by przemo (Scribe) on Apr 21, 2009 at 14:44 UTC
If visual solution for recording the session is acceptable, please take a look at Selenium. You can record the session in your browser and export it to Perl code.	[reply]