URAvgDeveloper101 has asked for the wisdom of the Perl Monks concerning the following question:

All, First off, This is my first posting here and I'd consider myself an intermediate Perl user at best, so don't be surprised by my ignorance on this. Now on to the question:

I have a script where I am posting a GET request using HTTP::Request and the LWP::UserAgent module. I get my response with:

$response = $ua->request($request);

I am able to print the content to stdout with:

print $response->content;

HOWEVER, I cannot print this to a file. When I try to print the contents to file I get an empty file. What I am really trying to do is access the content data from a calling C program with popen, but that does not work since nothing is going to stdout (at least from what I can tell).

I figured that if I am able to print the content to stdout as above then it is essentially, "What you see is what you get". I am inclined to believe I am missing something. Namely, some kind of encoding that Perl makes transparent to the user when they call a simple print command from within the script?

PLEASE help as I have been pulling my hair out half the day over this stupid thing and I know I am missing something relatively trivial. THANK YOU.

Replies are listed 'Best First'.
Re: Cannot access HTTP::Response content properly
by ikegami (Patriarch) on Nov 02, 2009 at 21:52 UTC

    First of call, calling ->content is pretty much always a bug. And it is here. You want

    print $response->decoded_content();

    or

    print $response->decoded_content( charset => 'none' );

    The latter won't undo the character encoding for XML and HTML docs.

    HOWEVER, I cannot print this to a file.

    That's unfortunate.

    I am inclined to believe I am missing something.

    Probably, but it's hard to tell what you're missing without knowing what you have.

    I am able to print the content to stdout

    that does not work since nothing is going to stdout

    uh, which one is it?

Re: Cannot access HTTP::Response content properly
by gmargo (Hermit) on Nov 02, 2009 at 22:06 UTC

    There's no reason you can't print the content to a file. Clearly something else is going on, probably with your popen business. Without some example code, there's little we can do.

    Can you distill your code down to a small example that demonstrates the problem? I bet you'll find the cause of the problem while trying to do so.

Re: Cannot access HTTP::Response content properly
by bichonfrise74 (Vicar) on Nov 02, 2009 at 23:10 UTC
    Is there a way you can post your code so that we can simulate the problem? I can test your code on my machine and if I am able to print it to a file, then at least it will narrow down your problem.

      All, Here is my code:

      #!/usr/bin/perl use strict; use LWP::UserAgent; my $ua; my $req; my $res; $ua = new LWP::UserAgent; $ua->agent("AgentName/0.1 " . $ua->agent); #REQUEST $req = HTTP::Request->new(GET =>'http://SOMEURLHERE'); #PASS REQUEST $res = $ua->request($req); #GET RESULTS if($res->is_success) { print $res->content; } else { print "No response\n"; print $res->status_line . "\n"; }

      Now the above code will print out just fine. However, if I try to add something like the following into the above code:

      open(MYFILE, '/tmp/data.txt'); print MYFILE $res->content; close(MYFILE);

      Or if I first assign the response data to a scalar and then do it:

      my $returnData = $res->content; open(MYFILE, '/tmp/data.txt'); print MYFILE $returnData; close(MYFILE);

      The file "/tmp/data.txt" is an empty file (although I didn't check it with "ls -l" which I probably should have). I just did a "cat" on the file and it was blank. I am also pretty sure that the popen command is not affecting things because if I print out a simple string from the same Perl script, that gets captured with popen. It has something to do with this HTTP::Response module.

        Extra tidbits:

        • If something isn't working as expected, one should start by checking what error occured! open, print and close all indicate if they were successful, and if not, why they weren't.

        • Declaring variables before they are used just introduces the possibility of errors. Instead of

          my $x; ... $x = ...;
          do
          ... my $x = ...;
        • Speaking of declaring variables, don't use global variables for file handles. That hasn't been necessary for 10 years.

          open(my $fh, '>', '/tmp/data.txt') or die("Can't create file /tmp/data.txt: $!\n"); print $fh $res->content; close($fh);
        • my $req = HTTP::Request->new(GET =>'http://SOMEURLHERE'); my $res = $ua->request($req);

          can be written as

          my $res = $ua->get('http://SOMEURLHERE');

        Your open statement

        open(MYFILE, '/tmp/data.txt');
        is opening the file for reading only. (See open.) Try this:
        open(MYFILE, '>', '/tmp/data.txt') || die("Cannot open /tmp/data.txt: +$!");