I haven't used your software before and did not have time to take a deep look at your page.. but it sounds like you want to do an awful lot. To say building "something like DBI" is a big job is an understatement. For example, why can you not use Net::FTP and other modules instead of reinventing them? Have you looked at inheriting from Net::Cmd?

Are you basically trying to implement file transfer over a bunch of different network services, and wrap it in a unified web interface?

You might like to look at IMS::ReleaseMgr which seems to do something similar to what you want to do.

There are also Penguin and One Penguin which claim to abstract away transport which could be email or whatever.

Maybe most useful would be to read LWP::Protocol module code. It does what you are talking about, finding what protocol is implemented, loading its module (just using require), blessing into a class, and dispatching a request and processing it. LWP/Protocol/http.pm shows one implementation. Actually you could look at LWP::Parallel::Protocol to answer your questions about scalability since you can run requests in parallel. Perhaps you would also like to look at LWP::Parallel::UserAgent.

From LWP::Protocol..

sub new
{
    my($class) = @_;

    my $self = bless {
        'timeout' => 0,
        'parse_head' => 1,
    }, $class;
    $self;
}      

sub create
{
    my $scheme = shift;
    my $impclass = LWP::Protocol::implementor($scheme) or
        Carp::croak("Protocol scheme '$scheme' is not supported");

    # hand-off to scheme specific implementation sub-class
    return $impclass->new($scheme);
}   

sub implementor
{
    my($scheme, $impclass) = @_;

    if ($impclass) {
        $ImplementedBy{$scheme} = $impclass;
    }
    my $ic = $ImplementedBy{$scheme};
    return $ic if $ic;

    return '' unless $scheme =~ /^([.+\-\w]+)$/;  # check valid URL sc
+hemes
    $scheme = $1; # untaint
    $scheme =~ s/[.+\-]/_/g;  # make it a legal module name

    # scheme not yet known, look for a 'use'd implementation
    $ic = "LWP::Protocol::$scheme";  # default location
    $ic = "LWP::Protocol::nntp" if $scheme eq 'news'; #XXX ugly hack
    no strict 'refs';
    # check we actually have one for the scheme:
    unless (@{"${ic}::ISA"}) {
        # try to autoload it
        eval "require $ic";
        if ($@) {
            if ($@ =~ /Can't locate/) { #' #emacs get confused by '
                $ic = '';
            } else {
                die "$@\n";
            }
        }
    }
    $ImplementedBy{$scheme} = $ic if $ic;
    $ic;
}
[download]

By the way the Freshmeat link on your page doesn't work, on purpose I suppose.

I guess it would be easiest if you could limit the number of functions your system has and find a least common denominator between the services you mention. I would guess all you really want would be file directory listing and file download, with your system simulating user sessions to remember the login/password combination of a given user. Hope this is on the same track as what you are thinking about.