in reply to mixing www::mech and lwp::UA

In the documentation for WWW::Mechanize you'll read, "WWW::Mechanize is a proper subclass of LWP::UserAgent and you can also use any of LWP::UserAgent's methods."

That means you don't need to explicitly 'use' and create an object instance of both. WWW::Mechanize @ISA LWP::UserAgent.

I wonder if the sites you're trying to scrape are intentionally rejecting your queries based on your user agent name? That's easy to test, by using $mech->agent_alias() to set your user agent name to something like Internet Explorer. Otherwise, the webserver sees you as something like LWP User Agent (if I recall). You could also check with the website's operator to see if they've got anything unusual going on there that might break your cookies. ...you are scraping within the TOS of the site, right? ;)


Dave

Replies are listed 'Best First'.
Re^2: mixing www::mech and lwp::UA
by Anonymous Monk on Nov 27, 2006 at 18:59 UTC
    Hi.

    I'm not exactly sure what that means, that I don't need to explicility use and create and object instance of both.

    From that, I imagined you meant that I could just call LWP::UserAgent and perform WWW::Mech functions without calling WWW::Mech. So from that, I tried

    $ua->agent_alias("Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) +");
    And it errors out saying there's no object method agent_alias.

    Do I have it backwards? If I call WWW::Mech, can I then use any LWP::UA methods I want?