Ray Smith has asked for the wisdom of the Perl Monks concerning the following question:

I'm, trying to better understand LWP::UserAgent cookie processing

Because of our requirements we must do initial processing using low-level HTTP operations, saving the cookie string, then create a LWP::UserAgent, create a HTTP::Request, add the cookie, and continue processing.

The short answer (to late...:) ) is that the subseqent call produce a 500 server error. In instrumenting my code with LWP::Debug I get the following when trying to set up the cookie and subsequent processing.

I appologize in advance for not being able to provide a complete program to demonstrate the problem. Also, sorry for the length of the description - I'm not quite sure where I am going wrong - cookie_jar setup, or subsequent call. Note it my other, low level code, I do fine in scanning for the cookie strings, which contain authentication information, and subsequently using the cookies for subsequent calls.

LWP coding: 

    my $ua = LWP::UserAgent->new();
    die "$userid: LWP::UserAgent->new() failed\n" if !defined($ua);
    my $r_reqlist = $ua->requests_redirectable();
    push @{$r_reqlist}, 'POST';
    my $cookiestr = $self->{cookies};
    if (defined($cookiestr)) {
    my $cookie_jar = $r_user->str2cookie_jar($cookiestr);
    $ua->cookie_jar($cookie_jar);
###    $request->header(Cookie=>$cookiestr);    # HACK ???
    }

    my $resp = $ua->request($request);
    $self->{responseCode} = $resp->code();
    $self->{responseMessage} = $resp->message();
    $self->{responseAsString} = $resp->as_string();

[download]

Cookie setup code cvts string obtained by scanning HTTP responses: I'm a little in the dark here, any further information on the process of setting up the cookie jar would be appreciated.

In guessing, I use the following values:

version: - 1

domain: - destination host's URL

path: - absolute path of file to be uploaded 

# Convert string to HTTP::Cookie object
# Only handles a single cookie for now
# If no header - create fake one: "Cookies:"
#  cookie := 
#       header_id ':' cookie_setting+
#       ;
#  cookie_setting := name '=' value ('; '| $)
#       ;
#  name := [^=\s]+
#       ;
#  value = [^"\s]* | '"' [^"]* '"'
#       ;
sub str2cookie_jar {    # Returns: HTTP::Cookies object
    my $self = shift;

    my (
    $cookieStr,              # as_string() representation
    ) = @_;

    my $cookie_jar = HTTP::Cookies->new();
    
    $self->log("str2cookie_jar: $cookieStr\n") if $DEBUG & 0x04;
    if ($cookieStr !~ /([^:]+):/) {
    $cookieStr = "Cookies:" . $cookieStr;
    }
    my ($cookie_header, $cookie_rest) = $cookieStr =~ /([^:]+):\s*(\S.
+*)/;
    die "Bad cookie_header in '$cookieStr'\n" if !defined($cookie_head
+er);
    die "Bad cookie body in '$cookieStr'\n" if !defined($cookie_rest);
    my %cookieh;
    my @cookiesettings = split(/;\s*/,$cookie_rest);
    my $version = $self->{cookieVersion};
    my $path = $self->{cookiePath};
    my $domain = $self->{cookieDomain};
    my $key;
    my $value;
    foreach my $cs (@cookiesettings) {
    if ($cs !~ /^([^=]+)=(.*)/) {
        if ($cs =~ /^\S+$/) {
        $key = $cs;           # Just presense
        $value = "";
        }
        else {
        $self->error("Bad cookie setting '$cs' in '$cookieStr'\n");
        return $cookie_jar;
        }
    }
    else {
        $key = $1;
        $value = $2;
    }
    
    if ($value =~ /^([\'\"])(.*)\1$/) {
        $value = $2;
    }
    $self->log("Cookie: $key=$value") if $DEBUG & 0x04;
    $cookie_jar->set_cookie($version, $key, $value,
                $path, $domain);
    }
    return $cookie_jar;
}

[download]

Run: 

 Wait for 5 seconds
Start IM crs01@edial.com crs02@edial.com
crs01@edial.com uploading Attachment file simpleAttachment.txt
callID: BIM_931_1139835130.033
LWP::MediaTypes::read_media_types: Reading media types from /usr/lib/p
+erl5/vendor_perl/5.8.5/LWP/media.types
LWP::MediaTypes::read_media_types: Reading media types from /home/rays
+mith/.mime.types
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
HTTP::Cookies::add_cookie_header: Checking qa04.edial.office for cooki
+es
HTTP::Cookies::add_cookie_header: - checking cookie path=/nfs/admin/pu
+blic/users/crs/testbuddy/simpleAttachment.txt
HTTP::Cookies::add_cookie_header:   path /nfs/admin/public/users/crs/t
+estbuddy/simpleAttachment.txt:/cgi-bin/buddies_attach does not fit
HTTP::Cookies::add_cookie_header: Checking .edial.office for cookies
HTTP::Cookies::add_cookie_header: Checking edial.office for cookies
HTTP::Cookies::add_cookie_header: Checking .office for cookies
LWP::UserAgent::send_request: POST https://qa04.edial.office/cgi-bin/b
+uddies_attach
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::collect: read 527 bytes
LWP::UserAgent::request: Simple response: Internal Server Error
Warning: crs01@edial.com: uploadAttachment POST https://qa04.edial.off
+ice/cgi-bin/buddies_attach %inputs failed
Response:
500
HTTP/1.1 500 Internal Server Error
Connection: close
Date: Mon, 13 Feb 2006 12:52:11 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Client-Date: Mon, 13 Feb 2006 12:52:12 GMT
Client-Peer: 192.168.80.104:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/ST=MA/L=Waltham/O=eDial/OU=auto-gen/CN=q
+a04.edial.office/emailAddress=replaceme@localhost
Client-SSL-Cert-Subject: /C=US/ST=MA/L=Waltham/O=eDial/OU=auto-gen/CN=
+qa04.edial.office/emailAddress=replaceme@localhost
Client-SSL-Cipher: AES256-SHA
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked
Title: 500 Internal Server Error

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>500 Internal Server Error</TITLE>
</HEAD><BODY>
<H1>Internal Server Error</H1>
The server encountered an internal error or
misconfiguration and was unable to complete
your request.<P>
Please contact the server administrator,
 xxx@yyy.com and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.<P>
More information about this error may be available
in the server error log.<P>
</BODY></HTML>


Request:
POST https://qa04.edial.office/cgi-bin/buddies_attach
User-Agent: libwww-perl/5.79
Content-Length: 313
Content-Type: multipart/form-data; boundary=xYzZY
Display: /content/ENG-US/htdocs/buddies/attachDialogInner.html
Mode: attach

--xYzZY
Content-Disposition: form-data; name="callID"

BIM_931_1139835130.033
--xYzZY
Content-Disposition: form-data; name="file_name"; filename="simpleAtta
+chment.txt"
Content-Length: 78
Content-Type: text/plain

simpleAttachment.txt

            Test File for
simpleAttachment.pl BuddiesUsers code.

--xYzZY--


upload attachment failed
  DB<1>

[download]

Janitored by Arunbear - replaced pre tags with code tags, and added READMORE tags.

Replies are listed 'Best First'.
Re: LWP ... cookie does not fit...
by jasonk (Parson) on Feb 13, 2006 at 15:13 UTC

    Have you tried looking in the web server error log, as the error message you have pasted here instructs? Generally a 500 error means the script crashed, and if it does, the error message that it crashed with would appear in the servers error log (or you can try adding use CGI::Carp qw(fatalsToBrowser); to try and get the errors to be displayed rather than logged).

    We're not surrounded, we're in a target-rich environment!
[reply]
[d/l]
      You hit the mark. Looking at the server log I saw a "Premature end of script headers" error. As it turns out, the server application is rather terse when it finds missing parameters. This lead me to doubt my handling of cookies with LWP. After supplying the proper parameters, things work fine. Thanks again.
[reply]

Back to Seekers of Perl Wisdom