Not sure about small changes. You obviously have a method of retrieving the data, so perhaps that method could look for similarities or patterns, then only retrieve the delta.

Its kinda had to give any concrete answer with out any specifics. If are able to provide an example of the data model it would make it easier to answer.

I wrote a startup script for mod_perl to
(1) set up a persistent database connection (in my case, MySQL),
(2) declare some global variables to use within the script.
(3) pre-load the script to be used by the outside world

Changes that will be referenced "every now and then" can be updated quickly in MySQL without problems. Remember to keep your tables small and properly indexed. The fewer UPDATE statements, the better. My final solution used one UPDATE statement, which meant the table was locked for a minimum amount of time without a chance of data corruption from an ill-timed SELECT statement by another child process.

Changes that must be immediately available to all can be stored in the global variable(s). Any future children will instantly have the correct info.

Doing this, I was able to maintain correct information without problems with up to 30 script calls per minute. No storage of session data was necessary. The script for the outside world becomes very small -- get the info, make UPDATEs if necessary, and spit out an answer...

-caedes

A persistent process could be a solution. Maybe Matts' pperl could do something for you.

Ciao!
--bronto

# Another Perl edition of a song:
# The End, by The Beatles
END {
  $you->take($love) eq $you->made($love) ;
}

Is there any way you can have 100% up-to-date data without doing the work to look it up? Not really. If there was, Oracle would be out of business. The closest you can get is the invalidation technique that I discussed above: every time anyone changes any of the data that you generate your responses from, you have to clear any cached responses that depend on that data. This is only possible in some situations.

I'd suggest you build it with no caching first, and then if it's slow you can look for where you'll get the most bang for your caching buck. You may find that your users will gladly trade a 5-minute delay on certain kinds of data for significantly better page response.

Also, think twice about putting all of that data into "session" objects. Unless it is specific to a single user, it doesn't belong in a session. Things like query results are not usually user-specific, i.e. a search for all stocks that have gone up in the 24 hours should be cached under "stocks, age < 24", not some session ID. In most systems users frequently repeat each other's queries, so caching results in a user-independent way gives better hit rates in the cache.