Are you basically trying to implement file transfer over a bunch of different network services, and wrap it in a unified web interface?

You might like to look at IMS::ReleaseMgr which seems to do something similar to what you want to do.

There are also Penguin and One Penguin which claim to abstract away transport which could be email or whatever.

Maybe most useful would be to read LWP::Protocol module code. It does what you are talking about, finding what protocol is implemented, loading its module (just using require), blessing into a class, and dispatching a request and processing it. LWP/Protocol/http.pm shows one implementation. Actually you could look at LWP::Parallel::Protocol to answer your questions about scalability since you can run requests in parallel. Perhaps you would also like to look at LWP::Parallel::UserAgent.

From LWP::Protocol..

sub new
{
    my($class) = @_;

    my $self = bless {
        'timeout' => 0,
        'parse_head' => 1,
    }, $class;
    $self;
}      

sub create
{
    my $scheme = shift;
    my $impclass = LWP::Protocol::implementor($scheme) or
        Carp::croak("Protocol scheme '$scheme' is not supported");

    # hand-off to scheme specific implementation sub-class
    return $impclass->new($scheme);
}   

sub implementor
{
    my($scheme, $impclass) = @_;

    if ($impclass) {
        $ImplementedBy{$scheme} = $impclass;
    }
    my $ic = $ImplementedBy{$scheme};
    return $ic if $ic;

    return '' unless $scheme =~ /^([.+\-\w]+)$/;  # check valid URL sc
+hemes
    $scheme = $1; # untaint
    $scheme =~ s/[.+\-]/_/g;  # make it a legal module name

    # scheme not yet known, look for a 'use'd implementation
    $ic = "LWP::Protocol::$scheme";  # default location
    $ic = "LWP::Protocol::nntp" if $scheme eq 'news'; #XXX ugly hack
    no strict 'refs';
    # check we actually have one for the scheme:
    unless (@{"${ic}::ISA"}) {
        # try to autoload it
        eval "require $ic";
        if ($@) {
            if ($@ =~ /Can't locate/) { #' #emacs get confused by '
                $ic = '';
            } else {
                die "$@\n";
            }
        }
    }
    $ImplementedBy{$scheme} = $ic if $ic;
    $ic;
}
[download]

By the way the Freshmeat link on your page doesn't work, on purpose I suppose.

I guess it would be easiest if you could limit the number of functions your system has and find a least common denominator between the services you mention. I would guess all you really want would be file directory listing and file download, with your system simulating user sessions to remember the login/password combination of a given user. Hope this is on the same track as what you are thinking about.

Yes, you have the gist of what I am saying. I want to expand what Web-FTP does to include other transports, and it seemed the best way to do this would be to write an interface module and release it, that way my code becomes much more useful, and maybe other people will write interfaces for it so I don't have to ;) I actually need a rather full set of features... login, listing, download, upload, various stats... basically my goal is to create an interface to Net::FTP and SSH (either through one of the SSH modules or with expect) that both work like Net::FTP (I think). I already have a vague idea on how to make the ssh work... later I would probably add a way to transfer from an mysql database, rsh, maybe others. I don't doubt that this will take some time, but I don't think doing the SSH and FTP ones should take forever... especially since the FTP one should almost just do a pass-thru to Net::FTP.

Looking at the two modules you pointed to don't seem to do exactly what I want, but the LWP stuff does look like a great place to start, thanks a lot.

I fixed the link to freshmeat, I never noticed that they changed the way they index their site... should have thought of it when they redesigned...

Thanks again

                - Ant