Re^2: Cannot access HTTP::Response content properly

All, Here is my code:

#!/usr/bin/perl

use strict;
use LWP::UserAgent;

my $ua;
my $req;
my $res;

$ua = new LWP::UserAgent;
$ua->agent("AgentName/0.1 " . $ua->agent);

#REQUEST

$req = HTTP::Request->new(GET =>'http://SOMEURLHERE');

#PASS REQUEST

$res = $ua->request($req);

#GET RESULTS


if($res->is_success)
{
   print $res->content;
}
else
{
   print "No response\n";
   print $res->status_line . "\n";
}
[download]

Now the above code will print out just fine. However, if I try to add something like the following into the above code:

open(MYFILE, '/tmp/data.txt');
print MYFILE $res->content;
close(MYFILE);
[download]

Or if I first assign the response data to a scalar and then do it:

my $returnData = $res->content;
open(MYFILE, '/tmp/data.txt');
print MYFILE $returnData;
close(MYFILE);
[download]

The file "/tmp/data.txt" is an empty file (although I didn't check it with "ls -l" which I probably should have). I just did a "cat" on the file and it was blank. I am also pretty sure that the popen command is not affecting things because if I print out a simple string from the same Perl script, that gets captured with popen. It has something to do with this HTTP::Response module.

Comment on Re^2: Cannot access HTTP::Response content properly Select or Download Code

Replies are listed 'Best First'.
Re^3: Cannot access HTTP::Response content properly by ikegami (Patriarch) on Nov 03, 2009 at 02:45 UTC
Extra tidbits: If something isn't working as expected, one should start by checking what error occured! `open`, `print` and `close` all indicate if they were successful, and if not, why they weren't. Declaring variables before they are used just introduces the possibility of errors. Instead of `my $x; ... $x = ...;` [download] do `... my $x = ...;` [download] Speaking of declaring variables, don't use global variables for file handles. That hasn't been necessary for 10 years. `open(my $fh, '>', '/tmp/data.txt') or die("Can't create file /tmp/data.txt: $!\n"); print $fh $res->content; close($fh);` [download] `my $req = HTTP::Request->new(GET =>'http://SOMEURLHERE'); my $res = $ua->request($req);` [download] can be written as `my $res = $ua->get('http://SOMEURLHERE');` [download]	[reply] [d/l] [select]
Re^4: Cannot access HTTP::Response content properly by Anonymous Monk on Nov 03, 2009 at 13:42 UTC
Hello All, Thank you so far. I took all your suggestions (e.g. decoded_content(), file handles, variable declarations, etc). It turns out though, that my hunch was right. I should have done an "ls -l" on the file and I would have discovered it was not an empty file. If you hadn't figured out already I am on a Unix/Linux system. Contrary to my original posting I thought I had done a "cat" when in actuality I had done a "more". Funny enough, regardless of whether I decode the content or not, something shows up with the "cat" command but not the "more" command. Also, I am able to edit the file in "vi" and see something. I am still stumped by the output though. Assuming that I know the contents of my file should be two lines of text separated by a newline. For example, "The Cat" and "The Dog" on separate lines: If I "cat" the file, I get: The Cat The Dog If I "vi" the file, I get: @^T@^h@^e@^ @^C@^a@^t@^ @^T@^h@^e@^ @^D@^o@^g@^ @^ Just for giggles, I did an "od -c", and I got: 0000000 \0 T \0 h \0 e \0 C \0 a \0 t \0 \n \0 T 0000020 \0 h \0 e \0 D \0 o \0 g \0 \n So for the $64 million questions, why is this showing up empty when I "more" the file, why is the calling program seeing it as empty as does the "more" command, and what's with these characters?	[reply]
Re^5: Cannot access HTTP::Response content properly by ikegami (Patriarch) on Nov 03, 2009 at 15:14 UTC
The file is encoded using UTF-16be	[reply]
Re^4: Cannot access HTTP::Response content properly by Anonymous Monk on Nov 03, 2009 at 15:17 UTC
If I decode this with UTF8, should that not take care of it?	[reply]
Re^5: Cannot access HTTP::Response content properly by ikegami (Patriarch) on Nov 03, 2009 at 15:34 UTC
It's not UTF-8, it's UTF-16be (or subset UCS-2be). Treating UTF-16be as UTF-8 is a bug. Output decoded text is a bug. If you decoded text, you need to encode it when you output it. You're free to use a different encoding, though.	[reply]
Re^6: Cannot access HTTP::Response content properly by Anonymous Monk on Nov 03, 2009 at 16:21 UTC
Re^7: Cannot access HTTP::Response content properly by ikegami (Patriarch) on Nov 03, 2009 at 17:57 UTC
Some notes below your chosen depth have not been shown here
Re^3: Cannot access HTTP::Response content properly by gmargo (Hermit) on Nov 03, 2009 at 00:53 UTC
Your open statement `open(MYFILE, '/tmp/data.txt');` [download] is opening the file for reading only. (See open.) Try this: `open(MYFILE, '>', '/tmp/data.txt') \|\| die("Cannot open /tmp/data.txt: +$!");` [download]	[reply] [d/l] [select]