Absolutely! I agree completely! The only reason i brought it up is because MySQL does not have a very elaborate cacheing mechanism and the thought of wastefulness always makes me cringe. Thus, I would liek to discuss DBI or CGI level cacheing. Perhaps I can be more specific about where and where not a CGI-level cacheing machanism would be appropriate.

One must consider, first of all, that the data may become dated if it is cached. For a search engine (especially a personal system) with one hour restarts for CGIs, cacheing is certainly an option. It's certainly not important that a new web page may not be displayed for an hour or so. One simple must make concessions and decisions on what is more important.

On the topic of system-persistent CGIs, I'd say it's the way to go in any case. I consider it like moving out of the Dark Ages. There is simply no reason not to have a CGI that can last some time with no memory leaks and a simple I/O interface.That's all really a CGI is anyway, a big function with HTTP client input. mod_perl and FastCGI both provide mechanisms for restarting CGIs and spawing CGIs as necessary or on a timed basis. A smarter CGI on a server that offers no such modern convenience could do something as simple as writing some text to the drive for its next run which may or not may be more efficient than stressing the dbengine.

Certainly cacheing will take up memory, but hell, for us, memory is cheap. If each page returned (with, for example, 10 entries) is 10K, then storing a hundred or a thousand of them will ease dbengine stress significantly while providing a faster response to the user, all under 10000K! If each page of results would require more RAM than you have, then cacheing is not a real possibility (ignoring virtual memory for a moment).

On the other hand, if what you're serving is tornado warnings, then you'll most likely not want to cache results for even 5 minutes. Web shopping is also probably not a good place to throw in cacheing unless you are cacheing certain parts of the results (and not, for example, the no. items available). In short, the cacheing scheme will need to depend on the urgency of the info, the space required for each cache (let's not forget compression schemes here, if necessary), as well as whether or not it's necessary at all! If the site doesn't get enough hits to warrant cacheing than one shouldn't waste his time hacking through it unless he expects the site to grow substantially quickly.

Even a more modern development of CGI may involve search branching to ensure that multiple CGI are not cacheing the same things. The CGI that grabs some info first, stores it for a configurable amount of time and other CGI processes that encounter the cached data request, know to pass over control to a different pool of CGIs that have the data already cached.

Sorry, chromatic, for the long discussion and actually I don't mean it to address it particularly towards or anyone else, I just suddenly had a stream of revelation that fit well here :-) The assembly lang. that I learned really helped me to think in terms of wasting as little as possible.

AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.