Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I need to make repeated spreadsheet downloads from a webpage. But since the data I need is daily, and I need it going back several years, this would take ages. I was told I could do this painlessly with Perl. I have only just learned the very basics of Perl. If someone could point me in the right direction it would be greatly appreciated. Thank you. Alex

Replies are listed 'Best First'.
Re: Automated downloads from a webpage
by Corion (Patriarch) on Jul 04, 2012 at 15:13 UTC

    The common way is to use one of the Mechanize modules for navigating websites, as these modules emulate a browser. For doing simple downloads without navigation, LWP::Simple is fairly simple.

Re: Automated downloads from a webpage
by zentara (Cardinal) on Jul 04, 2012 at 15:04 UTC
Re: Automated downloads from a webpage
by stevieb (Canon) on Jul 04, 2012 at 15:06 UTC

    In order to help, we'll need to know what your path looks like. For instance, are the dates organized into directories like this:

    example.com/2012/07/04/file.xls

    ...or like this:

    example.com/20120704-file.xls

    Give us a couple of examples of what your path looks like, and we can point you in the direction you need to be.

Re: Automated downloads from a webpage
by Anonymous Monk on Jul 04, 2012 at 15:58 UTC
    Thanks everybody. I'll try all this and let you know how it goes!
Re: Automated downloads from a webpage
by ig (Vicar) on Jul 05, 2012 at 09:07 UTC

    You might also consider the terms of use of the website. If you want several years worth of data, it may be simpler and more appropriate to contact the data owner and get it otherwise than by automated downloads of "daily" spreadsheets. If you do automate the download of a large quantity of data, you should take care not to overload the site. For example, you might limit the rate of your downloads.