Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Automated extraction of NOAA weather satellite images

by Scott7477 (Chaplain)
on May 29, 2007 at 22:25 UTC ( [id://618070]=CUFP: print w/replies, xml ) Need Help??

NOAA updates its weather satellite images on the GOES website every 30 minutes, with images in infrared, visible light, and water vapor wavelengths; for both the eastern and western halves of the US. The script below grabs those images and saves them to a local file with a timestamp. Could prove useful for doing your own weather forecasting or whatever....
use strict; use LWP::Simple; use Time::Tiny; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); my $current_time = Time::Tiny->now; my $timestring = $current_time->hour.$current_time->minute.$curren +t_time->second; print $timestring."\n"; foreach my $key (keys %images) { print $key."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };

Replies are listed 'Best First'.
Re: Automated extraction of NOAA weather satellite images
by graff (Chancellor) on May 30, 2007 at 02:40 UTC
    I had never heard of Time::Tiny before... If I understand its man page correctly, I think your code would create variable-length file names that might be hard to interpret reliably -- and might not be unique over the course of a day. That is, when you do:
    my $timestring = $current_time->hour.$current_time->minute.$current_ti +me->second;
    the time-stamp portion of each file name will range from "*.000.jpg" at midnight to "*.235959.jpg" at one second before midnight, with potential file-name collisions among sets of times like 01:23:45, 12:03:45 and 12:34:05 (which all would end up as "*.12345.jpg"), for example.

    Personally, I'd use the POSIX "strftime" function to set up fixed-width (zero-padded) strings for the file names -- and while I'm at it, might as well throw in the "ymd" date as well. Also, I think I'd rather use less repetition in the literal strings (looks like there was an inconsistency in your file names: an extra "v" in "westconusvwv").

    use strict; use LWP::Simple; use POSIX; my %image_url; for my $e_w ( qw/eastconus westconus/ ) { for my $typ ( qw/ir vs wv/ ) { $image_url{$e_w.$typ} = "http://www.goes.noaa.gov/GIFS/" . uc(substr($e_w,0,1)) . 'C' . $typ . '.JPG'; } } my $timestring = strftime( "%Y%m%d_%H%M%S", localtime ); print $timestring; for my $img ( keys %image_url ) { my $status = getstore( $image_url{$img}, join( '.', $img, $timestr +ing, 'jpg' )); print join( "\n", $img, $image_url{$img}, $status, "\n" ); }
    For that matter, it would be even simpler if I decided that the original four-letter NOAA file names were good enough for me...

    (updated the code to add a missing close-paren at line 16. Thanks, zentara!)

      Thanks for the feedback. I was looking at the POSIX module for including a timestamp in the saved filenames, but wanted to get a script up and running quickly and having never used the "strftime" function you mention Time::Tiny filled the bill. I agree with the potential problems you point out that the timestamps in my code have... I have updated my code to use the "strftime" function and the filenames now are very precise(for example "eastconusir20070530_074348.jpg"):
      #NOAA updates the images on the GOES website every 30 minutes use strict; use LWP::Simple; use POSIX; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); my $timestring = strftime( "%Y%m%d_%H%M%S", localtime ); print $timestring."\n"; foreach my $key (keys %images) { print $key."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };


Re: Automated extraction of NOAA weather satellite images
by jasonk (Parson) on May 30, 2007 at 13:37 UTC

    As it happens, I wrote a very similar script a couple of weeks ago. Rather than using the current time though, mine builds a timestamp from the Last-Modified header of the response, so the timestamp is the last time the image was updated, and you can run it as often as you want without downloading duplicate images.

    #!/usr/local/bin/perl -w ################## # Copyright 2007 Jason Kohles (JSK), <email@jasonkohles.com> ################## use strict; use warnings; use LWP::Simple qw( getstore head ); use POSIX 'strftime'; #chdir( "/home/weather/noaa-goes-images" ) or die "Cannot chdir: $!"; for my $loc ( qw( EC WC ) ) { for my $type ( qw( VS IR WV ) ) { my $url = "http://www.goes.noaa.gov/GIFS/".$loc.$type.".JPG"; my $epoch = ( head( $url ) )[ 2 ] || next; my $time = strftime( '%F-%T', gmtime( $epoch ) ); my $file = lc( join( '-', $loc, $type, $time ).'.jpg' ); next if -f $file; print "Retrieving $url to $file\n"; getstore( $url, $file ); } }

    We're not surrounded, we're in a target-rich environment!
      It's a good idea to timestamp the images with Greenwich Mean Time since that's what NOAA uses: I updated my script with your modified timestamp.

      #NOAA updates the images on the GOES website every 30 minutes use strict; use LWP::Simple; use POSIX; my $image; my $url; my %images = ( "eastconusir" => "http://www.goes.noaa.gov/GIFS/ECIR.JPG", "eastconusvis" => "http://www.goes.noaa.gov/GIFS/ECVS.JPG", "eastconuswv" => "http://www.goes.noaa.gov/GIFS/ECWV.JPG", "westconusir" => "http://www.goes.noaa.gov/GIFS/WCIR.JPG", "westconusvis" => "http://www.goes.noaa.gov/GIFS/WCVS.JPG", "westconusvwv" => "http://www.goes.noaa.gov/GIFS/WCWV.JPG", ); foreach my $key (keys %images) { print $key."\n"; my $epoch = ( head( $images{$key} ) )[ 2 ] || next; my $timestring = strftime( "%Y%m%d_%H%M%S", gmtime( $epoch ) ); print $timestring."\n"; print $images{$key}."\n"; my $status = getstore($images{$key},$key.$timestring.".jpg"); print $status."\n" ; };

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://618070]
Approved by graff
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2024-04-19 02:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found