pip has asked for the wisdom of the Perl Monks concerning the following question:

Blessings, Brothers,
My weather fetching script on my website no longer works. How it normally worked: html page generated form 'get' request invocing the script, which generated weather results, saved as file with js extension. Problem: Script still logs into bom (buero of met, australia), but is unable to fetch, or generate the correct data.
Any help is most appreciated...I tinkered around with it, but to no success :(

The script: 
#!/usr/bin/perl

use CGI::Carp qw(fatalsToBrowser);

open(W, "weather.js");
@lines = <W>;
close(W);
$out = join('', @lines);

if (scalar(@lines) < 3 or -M "weather.js" > 0.25) {
use LWP::UserAgent;

$ua = LWP::UserAgent->new();
$ua->timeout(60);
$request = HTTP::Request->new('GET', 'http://www.bom.gov.au/cgi-bin/wr
+ap_fwo.pl?IDS10034.txt');
$res = $ua->request($request);
if ($res->is_error) {
  $str = $res->message;
} else {
  $str = $res->content;
}

$str =~ s/^.*IDS1003401//s;
$str =~ s/HEADLINE.*?$//s;

if ($str =~ /:\s+(.*?)\n\n/s) {
  $adeldesc = $1;
  $adeldesc =~ s/\n//g;
}

if ($str =~ /Adelaide City\s+(.*?)\s+Max\s+(\d+)/s) {
  $adelmax = $2;
}

if ($str =~ /Noarlunga\s+(.*?)\s+Max\s+(\d+)/s) {
  $norlmax = $2;
}

$out =<<EOF;
var adeldesc = "$adeldesc";
var adelmax = "$adelmax";
var norlmax = "$norlmax";

EOF
open(W, ">weather.js");
print W $out;
close(W);
}
print <<EOF;
Content-Type: text/x-javascript
Pragma: no-cache
Cache-Control: no-cache
Expires: -1

$out
EOF

[download]
Desired output example, generated in weather.js: 
var adeldesc = "Fine, mild and sunny afternoon. Moderate southeast win
+ds and afresh afternoon sea breeze.";
var adelmax = "29";
var norlmax = "32";
var elizmax = "26"; (a new field, which I'd love to have)

[download]

Replies are listed 'Best First'.
Re: weather fetching script
by McDarren (Abbot) on Jul 13, 2006 at 05:26 UTC
    I'm in a generous mood today, so here is a re-write of your script that should work a bit better than the original: 
    #!/usr/bin/perl -w
use strict;

use HTML::TokeParser;
use LWP::UserAgent;

my %cities = (
        "City"
            => { var => 'adelmax' },
        "Elizabeth"
            => { var => 'elizmax' },
        "Noarlunga"
            => { var => 'noarmax' },
        "Mount Barker"
            => { var => 'mtbkmax' }
            );

my $ua = LWP::UserAgent->new();
$ua->timeout(60);
my $request = HTTP::Request->new('GET', 'http://www.bom.gov.au/cgi-bin
+/wrap_fwo.pl?IDS10034.txt');
my $result = $ua->request($request);
my $content = $result->content;

my $p = HTML::TokeParser->new(\$content);

my @lines;
while (my $tag = $p->get_tag("pre")) {
    my $text = $p->get_text("/pre");
    @lines = split /\n/, $text;
}

my $adeldesc;
my $cnt;

for my $line (@lines) {
    if ($line =~ /^Forecast for/ && !$adeldesc) {
        $adeldesc = $lines[$cnt+1];
    }

    if ($line =~ /^([\w\s]+):\s+Max\s(\d+)/) {
        $cities{$1}{max} = $2;
    }
    $cnt++;
}

my $out = "var adeldesc = $adeldesc\n";
for (keys %cities) {
    $out .= qq(var $cities{$_}{var} = "$cities{$_}{max}"\n);
}

open W, ">", "weather.js"
    or die "Could not open weather.js for writing:$!\n";
print W $out;
close W;

print <<EOF;
Content-Type: text/x-javascript
Pragma: no-cache
Cache-Control: no-cache
Expires: -1

$out
EOF

    [download]
    Output: 
    Content-Type: text/x-javascript
Pragma: no-cache
Cache-Control: no-cache
Expires: -1

var adeldesc = Fine and cool day with increasing high cloud.  Moderate
+ east to northeast winds.
var mtbkmax = "11"
var noarmax = "13"
var adelmax = "13"
var elizmax = "13"

    [download]

    As cwry pointed out, the reason it probably broke in the first place is because our good friends (and my ex-collegues) at the BOM decided to change "Adelaide City" to just "City". However, there were several other bad™ practices exhibited in your code, such as:

    • not using warnings and strict,
    • not checking the return value of your open,
    • using multiple scalar variables, where a hash would be more appropriate
    • and probably a couple more that I missed... ;)

    Cheers,
    Darren :)

[reply]
[d/l]
[select]
      Hi Darren, Thanks for the quick reply. The script fetches the description ok, but unfortunately not the actual temperatures. I can't get it to display the description in html, not sure if that is because of the missing enclosing tags "abcdef"; or the page itself. I'll do a mock js results page to fix that side of things. thanks again, pip
[reply]
        Okay, well that's because the content on layout of the page actually changes according to the time of day. Later in the afternoon they start including the forecast overnight minimums as well. Given that I used to work for them, I should have known this - but hey, it was a few years ago ;)

        Anyway, this is easy to get around. Given that you aren't interested in the minimums, you can simply alter the pattern match on line 40 to read as follows: 

        if ($line =~ /^([\w\s]+):.*?Max\s+(\d+)/) {

        [download]

        The change is that we include a non-greedy .* to skip over anything until it reaches "Max".

        With that change (and the current content as at 9:30PM CST), I get: 

        var adeldesc = Fine and cool with increasing high cloud.  Moderate eas
+t to northeast winds.
var mtbkmax = "9"
var noarmax = "11"
var adelmax = "10"
var elizmax = "11"

        [download]

        Cheers,
        Darren :)

[reply]
[d/l]
[select]
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: weather fetching script
by cwry (Monk) on Jul 13, 2006 at 05:07 UTC

    Hi,

    It looks to me like the BOM URL you are accessing ( http://www.bom.gov.au/cgi-bin/wrap_fwo.pl?IDS10034.txt) does not contain the data you are expecting. Presumably the BOM have modified the output format slightly, if the script was working in the past.

    Extracted from the URL above: 

    Precis        Fine. Cloud increasing.
City:         Max 13
Elizabeth:    Max 13
Mount Barker: Max 11
Noarlunga:    Max 13

    For example, you are searching for /Adelaite City\s+.../ but the appropriate search from the result looks to be something like /City:\s+Max (\d+)/. Likewise, you are searching for /Noarlunga\s+.../ but the relevant string would be something like /Noaralunga:\s+Max (\d+)/.

    I trust that from the above, you can work out how to modify your script appropriately?

[reply]
[d/l]
[select]
      Hi cwry, Thanks for the pointer, fiddled around with the script but can't seem to get it working. I am very much of a novice with perl scripts when it comes to repairs or mod's. I appreciate your help. cheers, pip
[reply]

Back to Seekers of Perl Wisdom