comment on

I have been trying to GET a page which requires a cookie to be received properly. What I eventually want to do is to POST some info to a form on this page and do a login with the help of LWP.

I know there are some other nodes on this subject, like this one LWP, extract_cookies, etc. , but I can't figure out what is wrong in my case.. The cookie which I receive does not seem to be "picked up" and is not saved to file (I save cookies in file for debugging purpose of this script).

What I bascily want to do is just to GET the url where the form, that I'm going to POST to later, is located. If I can receive the url with correct cookie handling, I'm sure POST'ing will not be difficult either. Here is my code:

#!c:/perl/bin/perl

  use strict;
  
  use LWP::UserAgent;
  use HTTP::Request;
  use HTTP::Cookies;
  use HTTP::Response;
  use HTTP::Headers;
  use HTTP::Message;
  use HTTP::Request::Common;
  use LWP::Debug;
  use CGI;
  
  #STORE COOKIES IN FILE
  my  $cookie_file  = qq(c:/tmp3/mycookie.txt);
  
  #HOW MANY TIMES HAVE $cookie_jar BEEN CHANGED 
  local $main::called_cookie = 0; 
  
  my  $query = new CGI;
  print $query->header;
  
  #NEW USER-AGENT OBJEKT
  my $ua = LWP::UserAgent->new;  
  $ua->agent('Mozilla/4.0');
  
  #NEW COOKIE_JAR 
  my $cookie_jar=HTTP::Cookies->new ('file' => "$cookie_file", 
 'autosave' => '1');
 
  #SUB TO BE CALLED WHEN $cookie_jar CHANGES _ FOR DEBUGGING 
  $cookie_jar->scan(\&cookiesub);
  
  $cookie_jar->clear();  
  
  #VARIOUS TEST-SITES WHERE WE RECEIVE COOKIES 
  #my $req  = GET 'http://www.cdnow.com';  
  #my $req  = GET 'http://www.msn.com' ;  
  #my $req  = GET 'http://www.amazon.com'; 
  
  #WANTED !!  
  my  $req  = GET 'http://mother.netcom.no/Login/login_page.html'; 
  
  #GO GET INFO
  my $resp=$ua->request($req); 
  
  #UPDATE $cookie_jar;
  $cookie_jar->extract_cookies($resp);  
  
  #PRINT RETURNED HEADERS 
  print $resp->headers_as_string();  
  
  #SHOW VALUE OF $cookie_jar NOW
  print "\n\nMy Cookie_jar is now:".$cookie_jar->as_string."\n\n"; 
  
  #SHOW RESPONSE
  #print  "\n\n".$resp->as_string;  
  
  #SAVE $cookie_jar TO FILE;
  $cookie_jar->save(); 
  exit;
  
  sub cookiesub {
  print "\n\n----------------\n\Cookie has been altered";
  print "\nCookie_counter:$main::called_cookie \n";
  $main::called_cookie++;
  print "@_";
  print "\n----------------\n";
  }
[download]

I receive reply that I need to "accept cookies" when I GET the wanted url. No cookie-values are saved to file, but when I print the cookie value with

print "\n\nMy Cookie_jar is now:".$cookie_jar->as_string."\n\n";
[download]

I get:

Set-Cookie3: NETSCAPE_LIVEWIRE_ID=305A68095915248EF4A4BD81DABFD4F90021
+A1F2; path="/Login"; domain="mother.netcom.no"; path_spec; discard; v
+ersion=0
[download]

What is wrong? Should this not have been handled automatic by LWP::UserAgent? What Can be done to receive the wanted URL with correct cookie handling?
I've tried doing several GET's after updating $cookie_jar with

$cookie_jar->extract_cookies($resp);
[download]

but that does not seem to help.

Also when I try to GET the amazon url, I see 3 Set-Cookies Headers in the HTTP::Response object ($res), but only 2 is saved to file? What is that all about?

I've been banging my head in the wall for too long about this. I need some help!

I'd REALLY appreciate any tips.
Thanks.
Regards, Tom

In reply to LWP cookie mystery by Anonymous Monk

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Pathologically Eclectic Rubbish Lister
	PerlMonks