Cookies with LWP (not gettin some)

GaijinPunch has asked for the wisdom of the Perl Monks concerning the following question:

Trying to emulate a couple of webpages to automate a task. I don't have the luxury of seeing the server side, so there's a lot of guess work. I'm basically filling out forms and either sending them through GET or POST. The login page basically keeps sending me back to the login page for some reason. I'm using Etheral to see what exactly is going on, doing in Firefox what the script is supposed to be doing. The first thing that I see is after the first post, 3 cookies get written when I'm in Firefox, but in my script, I only get two. My posting differences are of course the User-Agent, but my script does not send any of the following.

Accept:
Accept-Language:
Accept-Encoding
Accept-Charset;
Keep-Alive:
[download]

The next messages is a 302 Response Code (redirection). Both my script and Firefox are redirected to the correct place, but Firefox gets a 3rd cookie, which has an expiration date. My script, does not.

I do not know that this is the source of the problem, but it's the only difference thus far I can find. Considering I don't have access to the server side, there's not a lot I can get here other than creative suggestions. I tried looking at LWP at CPAN, but I couldn't find other ways to set those values that Firefox sends but lwp doesn't.

Comment on Cookies with LWP (not gettin some) Download Code

Replies are listed 'Best First'.
Re: Cookies with LWP (not gettin some) by Corion (Patriarch) on Mar 29, 2006 at 08:33 UTC
The problem is, of course, that you're not sending the same header/data as FireFox does. There is no magic solution by which we can tell you in what way the data your script sends differs from the data your browser sends. You will need to analyze the differences between the data - maybe it is already enough to use WWW::Mechanize instead of raw LWP, as WWW::Mechanize provides behaviour close to browsers, but maybe it isn't. You have to look at the differences. The Ethereal approach is the right direction - I've written myself Sniffer::HTTP to automate the sniffing of HTTP headers, and HTTP::Request::FromTemplate to regenerate requests from text files.	[reply]
Re^2: Cookies with LWP (not gettin some) by GaijinPunch (Pilgrim) on Mar 29, 2006 at 08:44 UTC
Thank you for that straight forward response. That was about as good as I could hope for in this situation I think. I'll have a look at Mechanize and see if that helps. Sniffer::HTTP sounds like it would be useful to!.	[reply]
Re: Cookies with LWP (not gettin some) by tirwhan (Abbot) on Mar 29, 2006 at 08:56 UTC
Further to Corion's response, you may also want to take a look at the Mozilla LiveHTTPHeaders Extension (which works with Firefox of course). I find this to be an invaluable and easy method for determining exactly what the browser and server are saying to each other. All dogma is stupid.	[reply] [d/l]
Re: Cookies with LWP (not gettin some) by Errto (Vicar) on Mar 29, 2006 at 20:33 UTC
Another way to track HTTP requests and responses if you don't want to (or for some reason can't) sniff raw TCP/IP traffic is to run a local instance of HTTP::Proxy and point both your browser and your script through that. Then take that proxy log and compare the results.	[reply]