Re^3: File caching for external storage

Exactly, server-side.

Squid is interesting I think, but I don't just want to proxy connections, I will need to do some programming.

One of the problems I have is that the PERL modules for S3 don't stream data, which means that if my application is sending a 2GB file to S3 then that 2GB file gets loaded into memory before it gets sent, that's bad. So I will store files over 500 MB locally and keep that info in a table.

But while I'm doing tables I might as well cache the most used files locally on my server instead of sending users to pick them up from S3 and incurring the bandwidth charge, say cache up to 100GB (the size of the local drive) before rotating out of the cache, and then how to determine which files should and shouldn't be in the cache.

This all seems to me to be something that somebody has to have done before, maybe not with S3, but just storing content and caching it. Maybe not...

Comment on Re^3: File caching for external storage

Replies are listed 'Best First'.
Re^4: File caching for external storage by tuxz0r (Pilgrim) on Nov 28, 2007 at 17:02 UTC
Cache::Cache looks like it has some of the functionality you need, specifically memory caching some of the smaller sizes or even caching on the local drive based on a size limit. I haven't used it before, so someone else can comment on its stability and their experiences with it. --- `echo S 1 [ Y V U \| perl -ane 'print reverse map { $_ = chr(ord($_)-1) } @F;'` Warning: Any code posted by tuxz0r is untested, unless otherwise stated, and is used at your own risk.	[reply]
Re^5: File caching for external storage by tcf03 (Deacon) on Nov 28, 2007 at 18:01 UTC
Cache::FileCache looks promising. Though, I have not used this myself. UPDATE scratch that. I should have read more. The Module code may prove interesting in your coding efforts. Ted -- "That which we persist in doing becomes easier, not that the task itself has become easier, but that our ability to perform it has improved." --Ralph Waldo Emerson	[reply]
Re^5: File caching for external storage by tosh (Scribe) on Nov 29, 2007 at 14:21 UTC
Yeah, the Cache::Cache stuff is really cool for the caching part of things, but it doesn't really seem to be very good for determining what should go IN or OUT of the cache, because obviously I will want more requested files going IN and less requested files coming OUT, but certainly it might be part of my application, thanks!!! OH!!!!! But Cache::SizeAwareCache might be exactly what I'm looking for, sweet!!!!	[reply]