in reply to Improve My Code...

The code below is used to create a list of dates from 18960101 to 20090717 using Date::Simple, access and submit a form using Mechanize
Just in principle, I'm going to object to this code on moral ground. That's roughly 45000 web hits, going through all the machinery to generate a web page for a non-human entity that will throw away 90% of what's generated. Even if it's your own machine (and if so, why bother with the web interface), it's a big waste of CPU. And if it's not your machine, it's unethical, unless you have written permission from the owners.

Just because you can scrape something doesn't mean you should. Please learn the difference.

-- Randal L. Schwartz, Perl hacker

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Replies are listed 'Best First'.
Re^2: Improve My Code...
by mchampag (Acolyte) on Aug 02, 2009 at 01:10 UTC
    According to the site's Statement of Purpose,
    We now have the capability to let the user choose a date of interest and get the weather statistics from that date. The entire record of daily observations is now available. Graphical representation of the data, especially long-term data, is included and is being augmented.

    cheech's use of the site is arguably reasonable. And maybe his hits will help them justify funding for more bandwidth or server horsepower--so he could be doing them a favor.

    Respectfully,
    Matt
[reply]
      It says "the user choose a date". If I was writing that, I'd have the expectation that humans would be entering a date, not a mechanized attack of all dates.

      True, there's nothing that forbids what he wants to do, but it's also a matter of courtesy. Don't go hammering someone else's machine without permission, and I wouldn't consider that paragraph permission, just description.

      -- Randal L. Schwartz, Perl hacker

      The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

[reply]
      Maybe they'll think about repackaging the data. Its static, organized by date, and a CGI interface isn't really the best way to deliver it.
[reply]
        Can you suggest a more practical alternative?
[reply]
Re^2: Improve My Code...
by Anonymous Monk on Aug 02, 2009 at 01:37 UTC
[reply]
Re^2: Improve My Code...
by cheech (Beadle) on Aug 02, 2009 at 01:59 UTC
    unless you have written permission from the owners.

    Go figure, I do. Read the op again, you didn't get what I was asking for I guess..

[reply]
Re^2: Improve My Code...
by Anonymous Monk on Aug 02, 2009 at 00:51 UTC
    FYI, it was made to be scraped. Please don't crusade at every opportunity.
[reply]

In Section Seekers of Perl Wisdom