Automated downloads from a webpage

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Automated downloads from a webpage by Corion (Patriarch) on Jul 04, 2012 at 15:13 UTC
The common way is to use one of the Mechanize modules for navigating websites, as these modules emulate a browser. For doing simple downloads without navigation, LWP::Simple is fairly simple.	[reply]
Re: Automated downloads from a webpage by zentara (Cardinal) on Jul 04, 2012 at 15:04 UTC
This useful article-> Automating-Web-based-Data-Retrieval-with-Perl ought to get you started. If you have difficulty, you can post your code here, and/or Google for perl Mechanize for more specific help. UPDATE: It seems IBM's javascript page links are broken. :-( In the meantime Google for perl WWW::Mechanize tutorial Update2: Here is a working link to the same Developer.com tutorial. -> Automating-Web-based-Data-Retrieval-with-Perl I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re: Automated downloads from a webpage by stevieb (Canon) on Jul 04, 2012 at 15:06 UTC
In order to help, we'll need to know what your path looks like. For instance, are the dates organized into directories like this: `example.com/2012/07/04/file.xls` ...or like this: `example.com/20120704-file.xls` Give us a couple of examples of what your path looks like, and we can point you in the direction you need to be.	[reply] [d/l] [select]
Re: Automated downloads from a webpage by Anonymous Monk on Jul 04, 2012 at 15:58 UTC
Thanks everybody. I'll try all this and let you know how it goes!	[reply]
Re: Automated downloads from a webpage by ig (Vicar) on Jul 05, 2012 at 09:07 UTC
You might also consider the terms of use of the website. If you want several years worth of data, it may be simpler and more appropriate to contact the data owner and get it otherwise than by automated downloads of "daily" spreadsheets. If you do automate the download of a large quantity of data, you should take care not to overload the site. For example, you might limit the rate of your downloads.	[reply]