Re: www::mechanize not behaving as browser?

WWW::Mechanize is behaving as a browser.

It is the stupid servers/cgi programs which is making assumptions, basically expecting a specific brand of browser and exploding when it gets something different.

So if you're going to fool these stupid programs, you have to pretend you are that specific brand of browser.

The way you do this is by using a regular browser to navigate the website successfully while you record the HTTP conversation with HTTP::Recorder/WireShark/Ethereal/LiveHTTPHeaders... and then you configure WWW::Mechanize to send similar headers....

Comment on Re: www::mechanize not behaving as browser?

Replies are listed 'Best First'.
Re^2: www::mechanize not behaving as browser? by jonnyfolk (Vicar) on Jul 24, 2010 at 10:54 UTC
Hi, thanks for your comment - I am taking a look at Wireshark now. I have tried several different browsers include IE & Firefox on Windows and Safari, Firefox and even Camino on Mac, and all have achieved successful search results. It doesn't seem too fussy about the particular browser, but there's obviously something different in my Mechanize browser which is sending things awry.	[reply]
Re^3: www::mechanize not behaving as browser? by Anonymous Monk on Jul 24, 2010 at 12:12 UTC
If it isn't filtering by the user-agent string, its probably setting a cookie via `img` (which mechanize doesn't load), or some form value via `script`... so it could be as simple as fetching an image, or set some extra form values normally set via `script` ... it all shows up on the wire :)	[reply] [d/l] [select]