Web Crawling problem with justDial

hemant.bhargava7 has asked for the wisdom of the Perl Monks concerning the following question:

Champs .. I just want to know that how to write a web crawler for justDial.com .. Actually i don't know how to pass values in those textboxes to next page .. i mean how i can automate that page .. I can automate eveything with Mechanize but Mechanize does not supports javascript .. So cant handle that also .. So if anybody can tell me even an inch of hint .. Then i'll be very thankful .. Cheers ..

Comment on Web Crawling problem with justDial

Replies are listed 'Best First'.
Re: Web Crawaler by ikegami (Patriarch) on Feb 16, 2009 at 10:30 UTC
This isn't a minor undertaking. Extracting links from HTML and CSS Fixing up relative links in files you save. Limits on depth and width of spidering. Obeying robot exclusion directives Throttling download rate and download request rate. Download from different servers at the same time Have you considered using existing tools?	[reply]
Re: Web Crawaler by poolpi (Hermit) on Feb 16, 2009 at 10:30 UTC
This example (from Randal SCHWARTZ) may help you : WWW::Mechanize::Examples hth, PooLpi 'Ebry haffa hoe hab im tik a bush'. Jamaican proverb	[reply] [d/l]
Re: Web Crawling problem with justDial by viren6524 (Initiate) on Aug 22, 2017 at 04:39 UTC
Just analyze api calls in chrome debugger network tab. Construct URL(add your required parameters like city, area, pageCount) fot api call and redirect to url. You will receive a json object, which contains results in json form, which you can use further.	[reply]