Re: Grabbing Data

One way to grab the output of a cgi script is to use URI::URL to construct the GET url, then LWP::UserAgent to fetch it;

#!/usr/bin/perl -w

eval 'exec perl -w -S $0 ${1+"$@"}'
    if 0; # not running under some shell

require 5;
use strict;
use LWP::UserAgent;
use URI::URL;
use HTTP::Request;

my $content;

my $ua = new LWP::UserAgent;

my $uri;

$uri = new URI::URL "http://172.16.4.25/cgi-bin/stats-cgi/hist.cgi";
$uri->query_form("netid" => 'system-SYSTEM',
                 "report_type" => 'Daily Totals',
                 "market" => 'PHL');

my $req = new HTTP::Request GET => $uri->as_string;

my $res = $ua->request($req);

if (!$res->is_success) {
    die "Fetch failed\n";
}
$content = $res->content;

print $content;
[download]

Comment on Re: Grabbing Data Download Code

Replies are listed 'Best First'.
RE: Re: Grabbing Data by ImpalaSS (Monk) on Oct 30, 2000 at 19:43 UTC
kilinrax, I pasted ran that code, and i got an error. The Internal Server 500 Error. I have no idea why, any ideas? Thanks again Dipul	[reply]
RE: RE: Re: Grabbing Data by kilinrax (Deacon) on Oct 30, 2000 at 20:10 UTC
That usually means the cgi script it's fetching isn't producing a valid header. Three obvious things to check: Is the url correct? Try printing `$uri` and exiting the script, and see what it's trying to fetch. Try checking your server logs ('`/var/log/apache/error.log`', for example) - what do they say? If you literally just c&p'd the code, check the shebang is pointing to where you have Perl installed, and that the modules are all there (see if '`perl -M'LWP::UserAgent' -V`' gives an error)	[reply]
RE: RE: Re: Grabbing Data by Fastolfe (Vicar) on Oct 30, 2000 at 20:14 UTC
They code he pasted is just a simple Perl script, not a CGI script. If you're trying to run this as a CGI script, you will need to modify it so it prints standard CGI headers. Most simply: `print "Content-type: text/html\n\n";` [download] should be sufficient. You may be interested in a book on writing Perl CGI scripts. Additionally, it's up to you to actually do something with this data. All the script does is print the content it retrieves straight back to the client/browser.	[reply] [d/l]
RE: RE: RE: Re: Grabbing Data by ImpalaSS (Monk) on Oct 30, 2000 at 20:48 UTC
Ok guys, this is what i did. I added a line at the end, print "works"; and then commented everything else out, the script worked. Then i started un-commenting lines. 1: require 5; (worked fine) 2: use strict; (worked fine) 3: use LWP::UserAgent; (500 Error) I hope this somewhat helps Thanks again, Dipul	[reply]
RE: RE: RE: RE: Re: Grabbing Data by Fastolfe (Vicar) on Oct 30, 2000 at 20:57 UTC
RE: RE: RE: RE: RE: Re: Grabbing Data by ImpalaSS (Monk) on Oct 30, 2000 at 21:02 UTC
Some notes below your chosen depth have not been shown here
RE: RE: RE: Re: Grabbing Data by ImpalaSS (Monk) on Oct 30, 2000 at 20:27 UTC
Hey guys, I tried everything Mentioned above, still gettin the 500 error. I changed the shebang line to where the perl is located, but still no luck. I went to look for the log, but again, i no luck. I went to /var/log but no apache directory?? Thanks guys Dipul	[reply]
(ar0n) RE (5): Grabbing Data by ar0n (Priest) on Oct 30, 2000 at 20:32 UTC
(jcwren) RE: (5) Grabbing Data by jcwren (Prior) on Oct 30, 2000 at 20:35 UTC
RE: RE: RE: Re: Grabbing Data by ImpalaSS (Monk) on Oct 30, 2000 at 20:33 UTC
Thanks again guys for the speedy replies. Ok, i did everything mentioned above yet i still get errors. I did change the shebang line to the one i used in my other scripts, but still no luck. I also added the mime line, but again, the 500 error. Then I tried looking for the error.log, but in the /var/log folder there was no apache folder, and the only logs in there weren't for errors. Any ideas? Dipul PS - I don't know if this helps, but when i went into the dir mentioned in the shebang line, /usr/local/bin/perl , i could get as far as usr/local/bin but there was no perl directory in there, just a perl@?	[reply]
RE: RE: Re: Grabbing Data by ImpalaSS (Monk) on Oct 30, 2000 at 20:10 UTC
One more question guys, Im not sure how to word this, but here goes. The code that kilinrax provided me, there is use LWP::UserAgent; use URI::URL; In it, how do i make sure this is installed on the server that i am running my perl on? Could that be the reason for the 500 error i am getting? Thanks Dipul	[reply]