tomazos has asked for the wisdom of the Perl Monks concerning the following question:

I want to get the content of a web page at a specific remote HTTP URL into a scalar.

I setup a script using LWP::UserAgent to do this, but it did not work because the server that the page is on requires the HTTP client to accept a cookie. It appears that LWP::UserAgent does not do this by default.

My code is as follows: 

use strict;
use warnings;
use LWP::UserAgent;

my $ua = new LWP::UserAgent;
$ua->agent("Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC)");
my $req = new HTTP::Request GET => 'http://groups.yahoo.com/group/some
+group/message/1243';

my $res = $ua->request($req);

if ($res->is_success) {
  my $pagecontent = $res->content;
} else {
 print "Bad luck this time\n";
}

[download]
Is it possible to tell LWP::UserAgent to accept a cookie? If not, is there another way of doing what I am trying to do?

Replies are listed 'Best First'.
Re: LWP::UserAgent and Cookies
by athomason (Curate) on Dec 12, 2001 at 02:06 UTC
    Sure, just set up a cookie jar using HTTP::Cookies: 
    use strict;
use LWP::UserAgent;
use HTTP::Cookies;
use HTTP::Request::Common;

my $cookie_jar = HTTP::Cookies->new( );
my $ua = LWP::UserAgent->new;

$ua->cookie_jar( $cookie_jar );

my $request = GET "http://www.doubleclick.net"; # don't worry, it does
+n't get saved :-)

my $response = $ua->request( $request );

$cookie_jar->extract_cookies( $response );

print $cookie_jar->as_string;

    [download]
    Subsequent requests using the same user agent will send those cookies to the server. If you don't want them to be sent, use a new agent or clear the jar with $cookie_jar->clear.

    When you say the server requires that the client accept a cookie, presumably you mean that the cookie must be kept around in order to retrieve another page correctly. Because when serving the first page, the server has no idea whether the user will keep the cookie (i.e. send it back later). If you mean it must remember it and send it back, you'll want to reuse the user agent (and cookie jar) as mentioned above. If you want to preserve the cookie jar for later use outside your script, take a look at the save and load methods in HTTP::Cookies. For a longer discussion of this subject, check out perldoc lwpcook and poke through the Q&A section under the HTTP and FTP clients topic.

[reply]
[d/l]
[select]
      Just a note:

      You don't need 

      
$cookie_jar->extract_cookies( $response );

      [download]
      If cookie jar object is set for user agent object than user agent does this itself.

      --
      Ilya Martynov (http://martynov.org/)

[reply]
[d/l]
        This is a very old thread, but having just tried myself I noticed that the cookie is not set if I don't make that method call.

        Although not needed in my example, I used a file for saving the cookie just to check the session-ID, and the file stayed empty without that call. Any thoughts?
[reply]
(ichimunki) Re: LWP::UserAgent and Cookies
by ichimunki (Priest) on Dec 12, 2001 at 02:10 UTC
    You might want to check out 'perldoc lwpcook', short but sweet examples of doing a bunch of different things with LWP.
[reply]
Re: LWP::UserAgent and Cookies
by Aighearach (Initiate) on Dec 12, 2001 at 02:15 UTC
    You need to use the $ua->cookie_jar method, which will require reading perldoc LWP::UserAgent and perldoc HTTP::Cookies.
    --
    Snazzy tagline here
[reply]
[d/l]

Back to Seekers of Perl Wisdom