in reply to Opening too many files?

That is a lot going on, and "high traffic" and "too much for a single page" are very subjective.

One simple technique is to cache the pages (or parts of the pages): as you're assembling the page the first time write it to a file or some other storage then send it out. On subsequent fetches of the same page, check to see how old the cached version is and if it's significantly old enough (fudge, fudge) rebuild it otherwise just chuck out the text in the cache.

You can keep this pretty lightweight if you remember to check the cache *before* you drag in all of the modules (incl. CGI, HTML::*, etc..). Then your overhead is simply the fork, exec of perl, and script compilation.

This is just a first approximation at scaling. In fact, you don't really have to do much at all to make this happen to existing scripts if they're written with this in mind.

Replies are listed 'Best First'.
Re(2): Opening too many files?
by FoxtrotUniform (Prior) on Jan 29, 2002 at 00:59 UTC

      One simple technique is to cache the pages (or parts of the pages): as you're assembling the page the first time write it to a file or some other storage then send it out. On subsequent fetches of the same page, check to see how old the cached version is and if it's significantly old enough (fudge, fudge) rebuild it otherwise just chuck out the text in the cache.

    Now, I know SFA about web caching, but wouldn't it make more sense to check the various components' last-modified times, and rebuild the page if any of them has changed (ala make). You'd only have to worry about "how long has it been since this page was fetched" if it's part of your cache replacement strategy.

    I was going to add something about this working best with a mostly static site, but I don't see how this would be worse than a timeout-based cache even for a fairly dynamic site. (Corrections are most welcome.)

    --
    :wq
[reply]
      Not a correction, a clarification perhaps.

      The original poster mentioned "forums" and "writeups". In a message system with threads, replies, etc.. all going on at the same time it might be almost as much trouble to find out if an article has replies that are newer than a pretermined mark than it would be to fetch the articles themselves.

      Unless, of course, it were built that way originally. From the OP's tone I gather it wasn't.

      Whereas assembling a page and presenting a recent but not completely dynamic view of the data wouldn't be harmful in the case of a message board.

      Using timestamps to determine if the static view should be rebuilt (or even having a background task doing it) isn't a bad strategy either if you can determine what your "timestamp" is.

[reply]

In Section Meditations