how can i give the key value in Scrappy crawl method

shivanisai has asked for the wisdom of the Perl Monks concerning the following question:

Look at the code given below.


 use Scrappy;

    my  $scraper = Scrappy->new;
    
        $scraper->crawl('http://search.cpan.org/recent',
            '/recent' => {
                '#cpansearch li a' => sub {
                    print $_[1]->{href}, "\n";
                }
            }


        );
[download]

Here '/recent' is given as a hash key value for the crawl method. I can understand it may be given from page_source (or) from the URL. But this is given only in scrappy 0.94111280,not in scrappy 0.62. I don't know how it should be given for other sites(For ex.http://www.google.co.in/).Could u please clear how it should be given?

Comment on how can i give the key value in Scrappy crawl method Download Code

Replies are listed 'Best First'.
Re: how can i give the key value in Scrappy crawl method by marto (Cardinal) on May 13, 2011 at 08:56 UTC
I believe the Google terms of service don't allow automated queries/screen scraping. If this is your goal I'd suggest using one of their APIs. For general web automation advice see my reply to another user yesterday, for a very similar (almost identical question) question.	[reply]
Re: how can i give the key value in Scrappy crawl method by Anonymous Monk on May 13, 2011 at 08:50 UTC
But this is given only in scrappy 0.94111280,not in scrappy 0.62. You can upgrade (install Scrappy 0.94111280) Could u please clear how it should be given? Try the 2nd example from the synopsis, use `->add` to add other urls to crawl.	[reply] [d/l]