wangcong has asked for the wisdom of the Perl Monks concerning the following question:

Hi, all!

I am wondering how I can cache the output of some commands invoked in my perl script? The reason why I do this is that every invoke of some command can be very expensive, so I don't want to run the commands everytime when I run my Perl script, I want to cache the result, and run the command only if the result is too out of date.

My idea is using temporary files for each command output, and use the time stamp of each file to know if that output is out of date or not.

Do you have any better ideas? Any comment will be appreciated.

Thanks!

Replies are listed 'Best First'.
Re: Cache commands output
by kyle (Abbot) on Oct 20, 2008 at 15:14 UTC
      Thanks for your reply, kyle.

      What I want to cache is the output of some command, not a file, so that is not what I want. :)

        You can certainly cache the output of some command with Cache::FileCache.

        sub qx_cache { my $cmd = shift; my $cache = Cache::FileCache->new({ namespace => $0 }); die 'no cache' if ! $cache; my $out = $cache->get( $cmd ); if ( ! defined $out ) { $out = qx{ $cmd }; $cache->set( $cmd, $out ); } return $out; }

        You can set a timeout value when you call set(). Any get() on a key that's expired will come back undef.

        I think kyle was right, what you want is Memoize

        This caches the result of a function by storring its return value in a hash, so the next time you call the function with the same parameters it will look up the return value in the hash and return that for you, rather then run the whole thing again.

        It also gives you the option to list the full hash so you can shove that in a file or database and load it again the next time you start your script resulting in lightning fast performance of a normally very slow fucntion, assuming you use the same input, which you must or caching makes no sense at all.

        I personally believe that this doesn't make sense, since the in root node you clearly say: "My idea is using temporary files for each command output, and use the time stamp of each file to know if that output is out of date or not." So basically it seems you want a form of caching/memoizing that is persistent across program invocations, and manages expiration dates. Cache::FileCache appears to do exactly this; in particular if you did care to read the very description "section" (it's one line!) of its documentation, then you would have found that:

        The FileCache class implements the Cache interface. This cache stores data in the filesystem so that it can be shared between processes.

        The additional emphasis is mine: please note that it talks about data which is agnostic wrt whered does it come from, be it a "command" or whatever...

        --
        If you can't understand the incipit, then please check the IPB Campaign.
Re: Cache commands output
by Illuminatus (Curate) on Oct 20, 2008 at 15:42 UTC
    It sounds like you want to store the results between runs of your perl script. In that case, what you suggest sounds reasonable. I would encapsulate this in either a function or a module, something like
    sub invokeCmd { my ($cmd) = @_; # assuming full path for cmd, change path from # cmd area to results area my $curResultsFile=$cmd."res"; # stat the file, check for existence, and 'last mod' value if ($too_old) { # re-run command and store results in $curResultsFile } my $res; open $res, $curResultsFile or die "could not open results: $!\n"; return $res; }
    Update What Kyle suggests would work just fine as well. Depending on how long your commands take to run, and how long you wish to store the results, You should probably avoid storing the results under /tmp (the default for Cache::FileCache), as a reboot deletes everything there.