in reply to Traversing directories to get the "most-recent" or "second-to-most-recent" directory contents

hacker,
This seems extremely straight forward.

Step 1:
To fetch the first page, I would probably use WWW::Mechanize provided that you comply with the robots.txt of the site.

Step 2:
To parse the directory listing page, I would use HTML::Parser, HTML::TokeParser::Simple, or HTML::TableContentParser - depending on what is appropriate.

Step 3:
The directories, if in the YYYYMMDD format as you describe, will sort correctly without the use of any module. Unless the listing was more than a few dozen entries - I wouldn't even bother with a modified water mark algorithm (I would just sort). If you do end up needing to use a date module, DateTime is the way to go.

Step 4:
Again, I would use WWW::Mechanize to go into the desired directory and get a contents listing. It should be easy to use the links() method to retrieve a list of files and then a simple regex to filter that list for files containing the date in question.

Step 6:
This is that hard part. I do not know of any modules on CPAN that allow you to resume downloads so you may have to implement this feature yourself. You could try WWW::Curl which is a wrapper module. In any case, this shouldn't be too difficult.

Cheers - L~R

  • Comment on Re: Traversing directories to get the "most-recent" or "second-to-most-recent" directory contents