Step 1:
To fetch the first page, I would probably use WWW::Mechanize provided that you comply with the robots.txt of the site.
Step 2:
To parse the directory listing page, I would use HTML::Parser, HTML::TokeParser::Simple, or HTML::TableContentParser - depending on what is appropriate.
Step 3:
The directories, if in the YYYYMMDD format as you describe, will sort correctly without the use of any module. Unless the listing was more than a few dozen entries - I wouldn't even bother with a modified water mark algorithm (I would just sort). If you do end up needing to use a date module, DateTime is the way to go.
Step 4:
Again, I would use WWW::Mechanize to go into the desired directory and get a contents listing. It should be easy to use the links() method to retrieve a list of files and then a simple regex to filter that list for files containing the date in question.
Step 6:
This is that hard part. I do not know of any modules on CPAN that allow you to resume downloads so you may have to implement this feature yourself. You could try WWW::Curl which is a wrapper module. In any case, this shouldn't be too difficult.
Cheers - L~R
In reply to Re: Traversing directories to get the "most-recent" or "second-to-most-recent" directory contents
by Limbic~Region
in thread Traversing directories to get the "most-recent" or "second-to-most-recent" directory contents
by hacker
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |