Re^2: Scrappy Module

Thanks for your immediate reply marto. I already worked with scrappy module. I can able to get the content when scrapping a website. My question is in the latest version of scrappy

 use Scrappy;

    my  $scraper = Scrappy->new;
    
        $scraper->crawl('http://search.cpan.org/recent',
            '/recent' => {
                '#cpansearch li a' => sub {
                    print $_[1]->{href}, "\n";
                }
            }
        );
[download]

you can find that this the url 'http://search.cpan.org/recent' means we need to give the 'recent' tag. it is working only for this cpan site. And it is not working for other sites. That is my question. How we could use the tags and get scrape a website. Can you able to understand.

Comment on Re^2: Scrappy Module Download Code

Replies are listed 'Best First'.
Re^3: Scrappy Module by marto (Cardinal) on May 12, 2011 at 11:48 UTC
I've never used this module, if you look at the source for the URL you provide in your example along with the documentation (see the `crawl` method), unless other sites contain the same '/recent' link and element with and id of 'cpansearch' etc, it's not going to work. In other words, you need to write your own code to work with your own sites. Other parsing modules are available, see WWW::Mechanize::Firefox among others.	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Scrappy Module
by marto (Cardinal) on May 12, 2011 at 11:48 UTC

I've never used this module, if you look at the source for the URL you provide in your example along with the documentation (see the crawl method), unless other sites contain the same '/recent' link and element with and id of 'cpansearch' etc, it's not going to work. In other words, you need to write your own code to work with your own sites. Other parsing modules are available, see WWW::Mechanize::Firefox among others.

[reply]
[d/l]