Pre-packaged perl CDN framework?

maruhige has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks

Evidently I'm not inquiring of the internet adequately, as I find it difficult to believe this hasn't been done before:

Is there such a thing as a pre-existing perl based, or perl interfaceable Content delivery network *framework*?

The context is a multi-site catalyst application which has a single application server, but multiple content servers. Specifically catalyst sits on the web and chucks one of two non-interactive apache services to resolve high bandwidth file requests. Due to shifting needs it would be valuable to have a content server inside the NAT barrier used by the client machines which is where I'm looking for what I can only describe as a CDN framework.

The existing system has some 10-15k folders with some .7m files distributed within. These are split about evenly between the two content servers with some overlap, with a master copy stored on the catalyst host. The decision as to which to provide to any given response is already handled by the application (although it could probably be done better).

What I need to do is to add in a couple of inner network hosts for a few hundred of the content folders which need to be updated with additions from the central master copy. I'd imagine this could be done with, e.g. rsync but the management of which blocks should be hosted inside should ideally be administered through the catalyst application in some way to reduce user training.

Any suggestions?

Comment on Pre-packaged perl CDN framework?

Replies are listed 'Best First'.
Re: Pre-packaged perl CDN framework? by Corion (Patriarch) on Jan 04, 2016 at 07:38 UTC
I would think about this problem in terms of rsync or maybe Varnish (or another web server) to distribute the master content. At least fastly.com uses Varnish scripting to move parts of the application into the edge servers. I'm wondering whether you couldn't just rsync all master content to all inner hosts and only serve the relevant parts of the data from them. This would reduce the administrative overhead as far as I understand it, as all inner hosts also have all data.	[reply]
Re^2: Pre-packaged perl CDN framework? by maruhige (Novice) on Jan 05, 2016 at 10:32 UTC
Varnish looks like a step in the right direction and possibly useful in peripheral activities. Regarding rsync/git/torrent to inner hosts this would almost certainly not be possible or even desirable with the resources available. The catalyst application provides searching, labeling and vast ammounts of meta data for the files, the sum total of which goes past the 800gb mark at last count (mainly proprietary format binaries) and then throws out a link to direct download. The value of the system is in the cat based meta data which needs central user/automated administration and update since it's practically the fulcrum of the entire workflow.	[reply]
Re: Pre-packaged perl CDN framework? by jellisii2 (Hermit) on Jan 04, 2016 at 14:14 UTC
Salt may be interesting to look at. It's designed to enforce state (which is what it sounds like you're trying to do) on machines. There are other solutions, but having worked with Salt, and using it currently, I can recommend it. It has some foibles, but generally it's very good.	[reply]