I would think there has to be some way to get a list of files in a remote directory without getting too carried away.

Not really. Not many servers are setup to allow directory indexing anymore. If going to the directory's URL in your browser shows you a list of files, then you can get the same list with a program of your own. If it doesn't, you can't. The other issue is that the list is normally returned as an HTML page, and different servers may format it in different ways, so it would be hard to create a consistent way to parse out the filenames. Since you're dealing with a single, known system, though, it might not be too hard in your case.

If the server will allow you to get the index of a directory, then there are various ways you could do it. You could fetch the URL with one of many HTTP aware modules like LWP::Simple or WWW::Mechanize. You could talk to the HTTP port directly through something like Net::Telnet. You'll still get an HTML page to deal with, though.

Another option would be to shell out to lynx, and use its -dump option to lose the HTML. My version of lynx returns the list of files like this:

     * [1]Parent Directory
     * [2]1
     * [3]2
     * [4]3
     * [5]abcd
[download]

So parsing that would be easy:

open my $in, "lynx -dump $myurl |" or die $!;
while(<$in>){
  chomp;
  if( /\[(\d+)\](.+)/ ){
    next if $1 == 1;
    print "$2\n";
  }
}
[download]