jeff_e has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to use LWP::UserAgent to get data from a simple web server I have created on an Arduino.

The request succeeds, but the decoded_content contains all of the response header. The information contained in the headers_as_string value is not sent from the server. The server sends "\r\n" for all line endings and an blank line with "\r\n" to mark the end of the header.

I have tried adding various attributes to the response header, but all had the same result of the header being in the decoded_content value.

Any thoughts on what may be causing the headers to be included in the content?

Thanks

Perl code 
#!/opt/local/bin/perl

use LWP::UserAgent;
use JSON;

use warnings;
use strict;


my $ua = LWP::UserAgent->new;

my $server_endpoint = "http://192.168.0.198/LED1";

my $req = HTTP::Request->new( GET => $server_endpoint );
$req->header('content-type' => 'application/json');


my $resp = $ua->request($req);
if ( $resp->is_success ) {
    my $header = $resp->headers_as_string;
    print "Received header:\n$header\n---------\n";
    my $message = $resp->decoded_content;
    print "Received reply:\n$message\n---------\n";

} else {
    print "HTTP GET error code: ", $resp->code, "\n";
    print "HTTP GET error message: ", $resp->message, "\n";
}

[download]
Output from above code 
Received header:
Client-Date: Mon, 17 Mar 2014 22:45:08 GMT
Client-Peer: 192.168.0.198:80
Client-Response-Num: 1

---------
Received reply:
HTTP/1.1 200 OK
Content-Type: application/json
Connection: close

{"val":1}

---------

[download]

Replies are listed 'Best First'.
Re: LWP UserAgent response header in decoded_content
by Anonymous Monk on Mar 18, 2014 at 09:13 UTC

    Any thoughts on what may be causing the headers to be included in the content?

    I think broken webserver

    For example

    use CGI();
my $q = CGI->new;
print $q->header, $q->header, $q->Dump;
__END__

    [download]
    This will show headers in the content ... but its just content in the content .. everything after header is content

    I'm not ruling out some odd /old misbehaving combination of LWP modules ... make sure you have the latest LWP...

    then fix your arudino server

    For diagnostics you might try using the following program and showing us the output 

    use Data::Dump qw/ dd /;
use WWW::Mechanize;
my $ua = WWW::Mechanize->new;
$ua->get( $server_endpoint );
dd( $ua );
__END__

    [download]
[reply]
[d/l]
[select]
      It's easy to see what the server returns to confirm this is a server problem. 
      perl -MIO::Socket::INET -e'
   my $s = IO::Socket::INET->new("192.168.0.198:80");
   print $s "GET / HTTP/1.0\r\n";
   print $s "Host: 192.168.0.198\r\n";
   print $s "\r\n";
   print while <$s>;
'

      [download]

      Note: I omitted Content-Type: application/json from the request since GET requests don't have a body (of any type).

[reply]
[d/l]
[select]

      I figured out the problem using the dd output. The server was sending null terminators from string literals. Removed those from the output of the server and LWP works as expected.

      Thanks for the suggestion and example.

      I figured it was something in the server output, but couldn't "see" anything wrong.

[reply]
Re: LWP UserAgent response header in decoded_content
by zentara (Cardinal) on Mar 18, 2014 at 10:06 UTC
    Another guess, is that maybe it is working correctly, and it is up to you to dereference whatever is contained in {"val".1} . Maybe the JSON module can do it? Why else do you have "use JSON;" in your code?
    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
[reply]

Back to Seekers of Perl Wisdom