GaijinPunch has asked for the wisdom of the Perl Monks concerning the following question:

Well, I think I kind of shot myself in the genetalia on this one. I had a working PHP script of a page that would login->parse->act->report within a Yahoo site. I wanted to make a Perl version, so got crackin'. Went all fine and dandy, until at some point today, they decided to up the security on me, forcing an anti-phishing image after I login.

I had a similar problem last night and as it turned out, I wasn't sending the proper Referer. Well, I am now, and I still get it. In fact, I tried logging in with Firefox and turning the Referer off (in Firefox) and I can log in no problem. So, they're detecing it some other way. Also note that the PHP script broke at this time, so they've basically blocked my IP, and likely any ID I've used from this IP.

Is there any other trick with Mechanize/Curl I'm missing? I'm heading to bed, but any nuggets you can offer while I sleep will be appreciated. This is driving me nuts. EDIT: It seems to be a combination of Referer + lack of cookie. If I wipe out my cookies and turn send referer off in Firefox, I'm foced w/ the image. If I delete the cookies but turn referring on, I'm not given the imge. So perhaps I'm not sending referrer as I thought I was. The other option is of course, to leave a cookie there.
  • Comment on WWW::Mechanize... some general scraping help

Replies are listed 'Best First'.
Re: WWW::Mechanize... some general scraping help
by Corion (Patriarch) on Sep 14, 2008 at 15:43 UTC
      I will check these threads out in the morning. Cheers.
Re: WWW::Mechanize... some general scraping help
by Mr. Muskrat (Canon) on Sep 19, 2008 at 20:35 UTC

    It sounds to me like Yahoo is trying to send you a message of "Stop doing that!"

    Are you sure that you are not in violation of their Terms of Service? (That is the one I get when I click the Terms of Service link on http://www.yahoo.com.)