Re: Grabbing Data
by kilinrax (Deacon) on Oct 30, 2000 at 19:27 UTC
|
One way to grab the output of a cgi script is to use URI::URL to construct the GET url, then LWP::UserAgent to fetch it;
#!/usr/bin/perl -w
eval 'exec perl -w -S $0 ${1+"$@"}'
if 0; # not running under some shell
require 5;
use strict;
use LWP::UserAgent;
use URI::URL;
use HTTP::Request;
my $content;
my $ua = new LWP::UserAgent;
my $uri;
$uri = new URI::URL "http://172.16.4.25/cgi-bin/stats-cgi/hist.cgi";
$uri->query_form("netid" => 'system-SYSTEM',
"report_type" => 'Daily Totals',
"market" => 'PHL');
my $req = new HTTP::Request GET => $uri->as_string;
my $res = $ua->request($req);
if (!$res->is_success) {
die "Fetch failed\n";
}
$content = $res->content;
print $content;
| [reply] [d/l] |
|
|
kilinrax,
I pasted ran that code, and i got an error. The Internal Server 500 Error. I have no idea why, any ideas?
Thanks again
Dipul
| [reply] |
|
|
That usually means the cgi script it's fetching isn't producing a valid header. Three obvious things to check:
- Is the url correct? Try printing $uri and exiting the script, and see what it's trying to fetch.
- Try checking your server logs ('/var/log/apache/error.log', for example) - what do they say?
- If you literally just c&p'd the code, check the shebang is pointing to where you have Perl installed, and that the modules are all there (see if 'perl -M'LWP::UserAgent' -V' gives an error)
| [reply] |
|
|
They code he pasted is just a simple Perl script, not a CGI script. If you're trying to run this as a CGI script, you will need to modify it so it prints standard CGI headers. Most simply:
print "Content-type: text/html\n\n";
should be sufficient. You may be interested in a book on writing Perl CGI scripts. Additionally, it's up to you to actually do something with this data. All the script does is print the content it retrieves straight back to the client/browser. | [reply] [d/l] |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
One more question guys,
Im not sure how to word this, but here goes.
The code that kilinrax provided me, there is
use LWP::UserAgent;
use URI::URL;
In it, how do i make sure this is installed on the server that i am running my perl on? Could that be the reason for the 500 error i am getting?
Thanks
Dipul
| [reply] |
(jcwren) RE: Grabbing Data
by jcwren (Prior) on Oct 30, 2000 at 19:26 UTC
|
I think we'd be more than happy to take a look at the HTML and propose a solution, but the IP address of the server you specified is for a private LAN (all 172.16.xx.xx addresses are).
Perhaps you could save a screen shot, and make it available on a publically accessible server.
Unless the data on the page is very horribly formatted, or subject to change format each time it loads, it's usually not too difficult to grab a page (LWP::Simple), then use HTML::Parser or HTML::TableExtract to grab data, and do something else with it. If the page you're grabbing data from is under your control, or you can persuade whomever it does belong to, making the same data available in XML format is one of the best solutions.
--Chris
e-mail jcwren | [reply] |
|
|
Chris,
Yeah sorry about that i completely forgot. Lemme grab a few of the lines in the output and just paste it in here :).
SYSTEM
Blocking IC_Call Handoff Intra BSC HO Inter BSC HO Loss Of Total
Traffic (Usage) in Erlangs Queue Rate DCCH Setup Failures Performance Performance Transmission Drop
Date DCCH PCH I.C. Disp Total Disp I.C. OvrHd Blkg Blkg L.O.R. WeakUL Drops Rate Drops Rate Drops Rate Events
today 887.5 24.22 3207 4187 8282 0.19% 1.15% 10.7% 0.03% 0.54% 2.65% 34.39% 600 0.57% 563 2.27% 2236 1.85% 3809
29Oct00 2563.0 71.55 14564 6433 23560 0.00% 0.01% 10.9% 0.14% 0.00% 0.05% 41.63% 2205 0.47% 2421 2.07% 9574 1.74% 16035
28Oct00 3349.9 92.96 20649 12320 36319 0.01% 0.07% 9.2% 0.47% 0.01% 0.29% 41.09% 3057 0.47% 3434 2.10% 13453 1.61% 22617
27Oct00 4733.1 396.76 39740 35833 80306 0.42% 2.34% 5.9% 0.23% 0.38% 6.20% 34.71% 7621 0.58% 8675 2.85% 28675 2.04% 50851
26Oct00 4907.0 401.84 38927 34829 78663 0.32% 1.89% 6.2% 0.08% 0.32% 4.81% 33.03% 7829 0.61% 8364 2.85% 30946 2.34% 54160
25Oct00 4564.6 549.88 38506 34345 77416 0.32% 2.03% 5.9% 0.02% 0.33% 5.04% 33.27% 6825 0.53% 7354 2.53% 24301 1.90% 43710
24Oct00 4580.0 458.84 37990 33411 75981 0.50% 2.54% 6.0% 0.10% 0.56% 6.76% 32.73% 6248 0.49% 7481 2.61% 23979 1.92% 42481
See he would want me to graph the DCCH, PCH, I.C., and DISP. I have no idea how to grab that data yet, but im working on it.
Also, how can i print an array, with a comma seperating every entry?
Thanks again everyone.
Dipul Patel | [reply] |
|
|
Again sorry about the triple post.
| [reply] |
|
|
Also, how can i print an array, with a comma seperating every entry?
print join(",", @array), "\n"
[ar0n]
| [reply] [d/l] |
|
|
sorry about the double (triple)post
| [reply] |
Re: Grabbing Data
by Fastolfe (Vicar) on Oct 30, 2000 at 19:28 UTC
|
The simplest answer is to use something like LWP::Simple to fetch the data, parse the output, and print the results as a normal CGI script, or a periodic script refreshing a static HTML page. | [reply] |
Re: Grabbing Data
by ImpalaSS (Monk) on Oct 30, 2000 at 19:35 UTC
|
Thanks Guys, Very Much. :)
Sorry about posting that data 3 times, i thought it was a preview, then i changed it, and it didn't look right, etc :)
Dipul | [reply] |