ImpalaSS has asked for the wisdom of the Perl Monks concerning the following question:

Hi, First off, let me ensure everyone that this is not homework. I am a college student on co-op with Nextel Communications :). This is my 6th week of perl, and i need some help. Basically my boss comes to me and says, "we need real time graphs of certian data". So i get some java applets that will graph the given data. I intend to write a perl script to actually input the data into the parameters of the applet. This is where my question comes in. I need the data grabbed from a html page that another perl script produces, but i need it so everytime the user wants a graph, it reruns the other script and grabs all of the data into an array. The URL that i need the data grabbed from is http://172.16.4.25/cgi-bin/stats-cgi/hist.cgi?netid=system-SYSTEM&report_type=Daily+Totals&market=PHL. Any help would be greatly appreciated
Dipul Patel

Replies are listed 'Best First'.
Re: Grabbing Data
by kilinrax (Deacon) on Oct 30, 2000 at 19:27 UTC
    One way to grab the output of a cgi script is to use URI::URL to construct the GET url, then LWP::UserAgent to fetch it;
    #!/usr/bin/perl -w eval 'exec perl -w -S $0 ${1+"$@"}' if 0; # not running under some shell require 5; use strict; use LWP::UserAgent; use URI::URL; use HTTP::Request; my $content; my $ua = new LWP::UserAgent; my $uri; $uri = new URI::URL "http://172.16.4.25/cgi-bin/stats-cgi/hist.cgi"; $uri->query_form("netid" => 'system-SYSTEM', "report_type" => 'Daily Totals', "market" => 'PHL'); my $req = new HTTP::Request GET => $uri->as_string; my $res = $ua->request($req); if (!$res->is_success) { die "Fetch failed\n"; } $content = $res->content; print $content;
      kilinrax, I pasted ran that code, and i got an error. The Internal Server 500 Error. I have no idea why, any ideas?
      Thanks again
      Dipul
        That usually means the cgi script it's fetching isn't producing a valid header. Three obvious things to check:
        1. Is the url correct? Try printing $uri and exiting the script, and see what it's trying to fetch.
        2. Try checking your server logs ('/var/log/apache/error.log', for example) - what do they say?
        3. If you literally just c&p'd the code, check the shebang is pointing to where you have Perl installed, and that the modules are all there (see if 'perl -M'LWP::UserAgent' -V' gives an error)
        They code he pasted is just a simple Perl script, not a CGI script. If you're trying to run this as a CGI script, you will need to modify it so it prints standard CGI headers. Most simply:
        print "Content-type: text/html\n\n";
        should be sufficient. You may be interested in a book on writing Perl CGI scripts. Additionally, it's up to you to actually do something with this data. All the script does is print the content it retrieves straight back to the client/browser.
        One more question guys,
        Im not sure how to word this, but here goes.
        The code that kilinrax provided me, there is
        use LWP::UserAgent;
        use URI::URL;
        In it, how do i make sure this is installed on the server that i am running my perl on? Could that be the reason for the 500 error i am getting?
        Thanks
        Dipul
(jcwren) RE: Grabbing Data
by jcwren (Prior) on Oct 30, 2000 at 19:26 UTC
    I think we'd be more than happy to take a look at the HTML and propose a solution, but the IP address of the server you specified is for a private LAN (all 172.16.xx.xx addresses are).

    Perhaps you could save a screen shot, and make it available on a publically accessible server.

    Unless the data on the page is very horribly formatted, or subject to change format each time it loads, it's usually not too difficult to grab a page (LWP::Simple), then use HTML::Parser or HTML::TableExtract to grab data, and do something else with it. If the page you're grabbing data from is under your control, or you can persuade whomever it does belong to, making the same data available in XML format is one of the best solutions.

    --Chris

    e-mail jcwren
      Chris, Yeah sorry about that i completely forgot. Lemme grab a few of the lines in the output and just paste it in here :).
      SYSTEM
                                                          Blocking                    IC_Call      Handoff      Intra BSC HO    Inter BSC HO       Loss Of      Total
                     Traffic (Usage) in Erlangs          Queue Rate         DCCH        Setup      Failures      Performance     Performance    Transmission     Drop
         Date    DCCH     PCH    I.C.    Disp   Total    Disp    I.C.   OvrHd   Blkg     Blkg  L.O.R.  WeakUL   Drops    Rate   Drops    Rate   Drops    Rate  Events
      today     887.5   24.22    3207    4187    8282   0.19%   1.15%   10.7%   0.03%   0.54%   2.65%  34.39%     600   0.57%     563   2.27%    2236   1.85%    3809
      29Oct00  2563.0   71.55   14564    6433   23560   0.00%   0.01%   10.9%   0.14%   0.00%   0.05%  41.63%    2205   0.47%    2421   2.07%    9574   1.74%   16035
      28Oct00  3349.9   92.96   20649   12320   36319   0.01%   0.07%    9.2%   0.47%   0.01%   0.29%  41.09%    3057   0.47%    3434   2.10%   13453   1.61%   22617
      27Oct00  4733.1  396.76   39740   35833   80306   0.42%   2.34%    5.9%   0.23%   0.38%   6.20%  34.71%    7621   0.58%    8675   2.85%   28675   2.04%   50851
      26Oct00  4907.0  401.84   38927   34829   78663   0.32%   1.89%    6.2%   0.08%   0.32%   4.81%  33.03%    7829   0.61%    8364   2.85%   30946   2.34%   54160
      25Oct00  4564.6  549.88   38506   34345   77416   0.32%   2.03%    5.9%   0.02%   0.33%   5.04%  33.27%    6825   0.53%    7354   2.53%   24301   1.90%   43710
      24Oct00  4580.0  458.84   37990   33411   75981   0.50%   2.54%    6.0%   0.10%   0.56%   6.76%  32.73%    6248   0.49%    7481   2.61%   23979   1.92%   42481
      

      See he would want me to graph the DCCH, PCH, I.C., and DISP. I have no idea how to grab that data yet, but im working on it. Also, how can i print an array, with a comma seperating every entry? Thanks again everyone. Dipul Patel
      Again sorry about the triple post.
        Also, how can i print an array, with a comma seperating every entry?

        print join(",", @array), "\n"

        [ar0n]

      sorry about the double (triple)post
Re: Grabbing Data
by Fastolfe (Vicar) on Oct 30, 2000 at 19:28 UTC
    The simplest answer is to use something like LWP::Simple to fetch the data, parse the output, and print the results as a normal CGI script, or a periodic script refreshing a static HTML page.
Re: Grabbing Data
by ImpalaSS (Monk) on Oct 30, 2000 at 19:35 UTC
    Thanks Guys, Very Much. :)
    Sorry about posting that data 3 times, i thought it was a preview, then i changed it, and it didn't look right, etc :)
    Dipul