I think that your problem is different. We already have a caching mechanism with LWP::Simple::mirror. What you are asking for is something that will defeat the purpose of HEAD method for something already stored somewhere and that will not obey no-cache instructions. This is reasonable, especially when you don't want to overload an external resource. I thought many times about this kind of caches and tried few times to write a module.

I came up with a 'not_before sometime' policy that is well is applicable to Cache::Cache mentioned by PodMaster; furthermore that module can be used to cache complex URLs. If you are interested I can post some code.

At the other end of the spectrum there are proxies, they are built to do this kind of work.

Ciao, Valerio

update: On second thought, transparently handling things sounds good, go for it.

MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"

I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).

** The third rule of perl club is a statement of fact: pod is sexy.

I know you already know all of this, but i replied to you because it seemed like the place. ;) I just had to take a stab at this:

use strict;
use warnings;

use LWP::Simple;
use Cache::MemoryCache;

my $cache = Cache::MemoryCache->new({
   namespace          => 'MyNamespace',
   default_expires_in => 60,
});


for (1..3) {
   my $page = $cache->get('perlmonks');
   unless ($page) {
      warn "fetching from web\n";
      $page = get('http://perlmonks.org');
      $cache->set('perlmonks', $page, "1 minute" );
   } else {
      warn "fetching from cache\n";
   }
}
[download]

The for loop is just to actually show that subsequent fetches (after the first) come from the memory cache and not the web. Use Cache::FileCache if you want persistence past the life of script execution.

Honestly, as easy as this is ... it's still 'plumbing'. I would like to see a 'LWP::Cache' module as long as it is a transparent wrapper around LWP and Cache::Cache. Why not? ;)

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

Something to think about...

If you store the Last-Modified and Etag header items and send them along in your request (as If-Modified-Since and If-None-Match respectively), a 304 code will be returned if the page hasn't changed. It should minimize some traffic for pages that don't change too often.

--
"To err is human, but to really foul things up you need a computer." --Paul Ehrlich

But instead of doing that, you could also just set up Apache's mod_proxy to be a caching proxy, and then point your UserAgent's proxy settings at your apache server. That'd effectively be the same thing.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

$will->code for @food or $$;