BUU has asked for the wisdom of the Perl Monks concerning the following question:

My basic idea here is, I'm going to be creating a website that will store a number of very large files (think 200 megs) and hopefully generate a fair amount of traffic. The problem here is that I can not afford to pay for a box/bandwidth large enough to provide ample service. So the idea is to enlist other websites and have them provide the content to save me bandwidth and storage.

So my question is, how can I best accomplish this using perl? The basic idea is to have a database of 'host websites', then whenever a file is uploaded my my server, it would choose say, three websites that still had available storage space and bandwidth, and then send the file to them for storage. I would then reccord the location of the file in another database, along with its size and so forth. Then when someone requests a file, I would select from the database of hosts that contain the file and still has bandwidth, then subtract the file size from the hosts bandwidth and redirect to that file.

So again my question, how can i do this with perl? I'm more the familiar with the database portion of this, but i'm unsure of the best way to replicate the file out to the other hosts. Some sort of lwp::* operation? Net::Ftp? But I'm also concerned about security, as well as over the ease of setting it up on the other hosts.

Replies are listed 'Best First'.
Re: Multiwebsite Distributed Content?
by valdez (Monsignor) on Mar 04, 2003 at 14:51 UTC

    Interesting question! I don't think you can save storage, because you may need to distribute every file to every possible site depending on what box has more traffic available. You can distribute files using whatever you want (FTP, HTTP, RSYNC), preferably something that can resume interrupted transfers, to avoid waste of traffic.
    On the other side, you must account also interrupted users' transfers, and this may be quite complex because you have to analyze logs, update traffic accordingly and give this informations back to your central repository.

    There are a couple of projects that you may find interesting: BitTorrent and Open Content Network.

    Ciao, Valerio

Re: Multiwebsite Distributed Content?
by zby (Vicar) on Mar 04, 2003 at 14:32 UTC
    For distribution you can try one of the rsync modules. Rsync is fast when updating files - since it tries to send only the differences.