Re: Pre-packaged perl CDN framework?

I would think about this problem in terms of rsync or maybe Varnish (or another web server) to distribute the master content. At least fastly.com uses Varnish scripting to move parts of the application into the edge servers.

I'm wondering whether you couldn't just rsync all master content to all inner hosts and only serve the relevant parts of the data from them. This would reduce the administrative overhead as far as I understand it, as all inner hosts also have all data.

Comment on Re: Pre-packaged perl CDN framework?

Replies are listed 'Best First'.
Re^2: Pre-packaged perl CDN framework? by maruhige (Novice) on Jan 05, 2016 at 10:32 UTC
Varnish looks like a step in the right direction and possibly useful in peripheral activities. Regarding rsync/git/torrent to inner hosts this would almost certainly not be possible or even desirable with the resources available. The catalyst application provides searching, labeling and vast ammounts of meta data for the files, the sum total of which goes past the 800gb mark at last count (mainly proprietary format binaries) and then throws out a link to direct download. The value of the system is in the cat based meta data which needs central user/automated administration and update since it's practically the fulcrum of the entire workflow.	[reply]

Replies are listed 'Best First'.

Re^2: Pre-packaged perl CDN framework?
by maruhige (Novice) on Jan 05, 2016 at 10:32 UTC

Varnish looks like a step in the right direction and possibly useful in peripheral activities.

Regarding rsync/git/torrent to inner hosts this would almost certainly not be possible or even desirable with the resources available. The catalyst application provides searching, labeling and vast ammounts of meta data for the files, the sum total of which goes past the 800gb mark at last count (mainly proprietary format binaries) and then throws out a link to direct download. The value of the system is in the cat based meta data which needs central user/automated administration and update since it's practically the fulcrum of the entire workflow.

[reply]