in reply to mixing www::mech and lwp::UA
In the documentation for WWW::Mechanize you'll read, "WWW::Mechanize is a proper subclass of LWP::UserAgent and you can also use any of LWP::UserAgent's methods."
That means you don't need to explicitly 'use' and create an object instance of both. WWW::Mechanize @ISA LWP::UserAgent.
I wonder if the sites you're trying to scrape are intentionally rejecting your queries based on your user agent name? That's easy to test, by using $mech->agent_alias() to set your user agent name to something like Internet Explorer. Otherwise, the webserver sees you as something like LWP User Agent (if I recall). You could also check with the website's operator to see if they've got anything unusual going on there that might break your cookies. ...you are scraping within the TOS of the site, right? ;)
Dave
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^2: mixing www::mech and lwp::UA
by Anonymous Monk on Nov 27, 2006 at 18:59 UTC | |
by Joost (Canon) on Nov 27, 2006 at 20:16 UTC |