Re^5: Scrappy user

Look at the source https://metacpan.org/source/AWNCORP/Scrappy-0.94112090/lib/Scrappy/Scraper/UserAgent.pm, first parameter is browser, second parameter the operating system, i.e. you would use something like scraper->user_agent("opera","Macintosh")

PS: As the others have said, if the sites doesn't want robots, you shouldn't just ignore that.

Comment on Re^5: Scrappy user_agent error Download Code

Replies are listed 'Best First'.
Re^6: Scrappy user_agent error by docster (Novice) on Jan 06, 2012 at 16:51 UTC
Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job.. `Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./scrappy.pl line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = 'http://google.com'; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print www.google.com __END__` [download]	[reply] [d/l]

Replies are listed 'Best First'.

Re^6: Scrappy user_agent error
by docster (Novice) on Jan 06, 2012 at 16:51 UTC

Can't locate object method "user_agent" via package "scraper" (perhaps
+ you forgot to load "scraper"?) at ./scrappy.pl line 8.


#!/opt/local/bin/perl
use strict;
use warnings;
use Scrappy;

my $url = 'http://google.com';
my $scraper = Scrappy->new;
scraper->user_agent("opera","Macintosh");
$scraper->get("$url");
print $scraper->domain, "\n"; # print www.google.com

__END__
[download]

[reply]
[d/l]