The code below submits a form which returns a page of weather data. I need the pages from every date available up to the current date, and the first available date is January 1 of 1896. So I loop over each date, submit the form, write the results page to a file, go back a page, and do it again for the next date..

However, for some reason I keep getting this error once I get to the the date 18961110 (November 10 of 1896).

Error GETing http://bub2.meteo.psu.edu/wxstn/wxstn.htm: Can't connect to bub2.me teo.psu.edu:80 (connect: timeout) at C:\Perl\scripts\i.pl line 19
The data for this date is available and I can access it manually by logging on to the site, so it's not that the server doesn't recognize the date.

What could be causing my program to fail at the same date each time? Also, it took about 3 minutes to get through October of the first year, yet I have over 100 years to go through. Is there any way for me to improve this code so that it was run faster?

Thanks a lot for any suggestions or advice!

use strict;
use warnings;

open(my $in, '<', "c:/perl/scripts/dates/dates.txt")
    or die "open: $!\n";

my @date;
while (<$in>) {
    chomp;
    push(@date, $_);
}
close($in);

use WWW::Mechanize;
foreach my $date (@date) {
    my $mech = WWW::Mechanize->new();
        $mech->get( "http://bub2.meteo.psu.edu/wxstn/wxstn.htm" );
        $mech->form_number(1);
        $mech->field( 'dtg' , $date );
        $mech->click();
        $mech->content();
    my $file = "c:/perl/scripts/dates/$date.txt";
        $mech->save_content($file);
        $mech->back();
    }

[download]

In reply to Connection Timeout duing form submissions by cheech

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.