Re^3: Scrappy user

Replies are listed 'Best First'.
Re^4: Scrappy user_agent error by docster (Novice) on Jan 03, 2012 at 18:45 UTC
Yes, it is entirely possible. I took the code above from the authors blog post. It is hard to find examples of a working Scrappy script. But as of now I am using: `Scrappy (0.94112090).` And CPANs Module Version: 0.94112090 docs from: http://search.cpan.org/dist/Scrappy/lib/Scrappy.pm#user_agent user_agent The user_agent attribute holds the Scrappy::Scraper::UserAgent object which is used to set and manipulate the user-agent header of the scraper. `my $scraper = Scrappy->new; $scraper->user_agent;` [download] So in that context, how would I set the user_agent correctly to be firefox using Scrappy 0.94112090? There used to be way. Maybe it was removed. I seem to be missing the entire picture somehow :)	[reply] [d/l] [select]
Re^5: Scrappy user_agent error by jethro (Monsignor) on Jan 04, 2012 at 10:16 UTC
Look at the source https://metacpan.org/source/AWNCORP/Scrappy-0.94112090/lib/Scrappy/Scraper/UserAgent.pm, first parameter is browser, second parameter the operating system, i.e. you would use something like `scraper->user_agent("opera","Macintosh")` PS: As the others have said, if the sites doesn't want robots, you shouldn't just ignore that.	[reply] [d/l]
Re^6: Scrappy user_agent error by docster (Novice) on Jan 06, 2012 at 16:51 UTC
Tested this but it still gives an error. Maybe Scrappy is just not the right tool for this job.. `Can't locate object method "user_agent" via package "scraper" (perhaps + you forgot to load "scraper"?) at ./scrappy.pl line 8. #!/opt/local/bin/perl use strict; use warnings; use Scrappy; my $url = 'http://google.com'; my $scraper = Scrappy->new; scraper->user_agent("opera","Macintosh"); $scraper->get("$url"); print $scraper->domain, "\n"; # print www.google.com __END__` [download]	[reply] [d/l]
Re^5: Scrappy user_agent error by roboticus (Chancellor) on Jan 04, 2012 at 04:01 UTC
docster: I've not used Scrappy, but perhaps you could check out the tests for examples on how to change the user agent name. ...roboticus When your only tool is a hammer, all problems look like your thumb.	[reply]