"AFAIK, it is entirely up to the server and the applications running on it to decide how and if a directory is rendered at any given URL. Just because I can get a document at http://someurl.com/documents/mydoc.txt doesn't mean I can get a directory at http://someurl.com/documents/. I might."

I inferred from his question (where he said, "http - standard apache index") that this was already not a problem. That he has a particular directory in mind which has a known directory listing format.

"Which module would you suggest for reliably getting the contents of a remote directory via HTTP?"

Personally I'd use Web::Magic, but I'm biased.

use 5.010;
use strict;
use PerlX::MethodCallWithBlock;

use Path::Class qw(file dir);
use Web::Magic -sub => 'web';
use XML::LibXML 2.0000;

my $listing     = URI->new('http://buzzword.org.uk/2012/');
my $destination = dir('/home/tai/tmp/downloaded/');

# Make sure destination directory exists.
$destination->mkpath;

web($listing)

    # Die if 404 or some other error
    -> assert_success
    
    # Find all the links on the page
    -> querySelectorAll('a[href]')
    
    # Skip uninteresting links
    -> grep {
        not (
            /Parent Directory/
            or $_->{href} =~ m{\?}  # has a query
            or $_->{href} =~ m{/$}  # ends in slash
        )
    }
    
    # Expand relative URI references to absolute URIs
    -> map {
        URI->new_abs($_->{href}, $listing)
    }
    
    # Save each to the destination directory
    -> foreach {
        # Figure out name of file to save as
        my $filename = $destination->file( [$_->path_segments]->[-1] )
+;
        
        # Log a message
        printf STDERR "Saving <%s> to '%s'\n", $_, $filename;
        
        # Save it!
        web($_)->save_as("$filename");
    }

[download]
perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'

In reply to Re^3: Remote Directory listing by tobyink
in thread Remote Directory listing by johnfl68

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.