in reply to CGI: Waiting for Output

If the hundreds of megabytes of flat text are relatively static (only changing once a week, say, or not at all), then the best thing would be to figure out how to structure it into something other than a flat text file -- either a dbm file as suggested by Zaxo, or a relational database.

If you're parsing through something like a log file that changes (grows) continuously, you could still consider creating "digests" at regular intervals that would not be flat text (i.e. searching/indexing would be relatively quick and properly focused from the user's perspective), so that a separate cron job handles the heavy lifting, and produces a derived data set that allows the browser interaction to rely on a simple/quick task.

In any case, having each browser connection produce its own distinct summary web page shouldn't be a problem -- just apply a file name convention that makes it easy for yesterday's html files to be deleted at noon or whatever.

Replies are listed 'Best First'.
Re: Re: CGI: Waiting for Output
by Anonymous Monk on Sep 02, 2003 at 03:07 UTC
    This is a good strategy, but not one that gets at the core of the problem. Let's say my script isn't parsing hundreds of megabytes of flat text (that's out of my control unfortunately, or else I would put it in a db as Zaxo suggests). Maybe instead it's performing some huge computation or doing some massive en/decryption or waiting for a reply from another server somewhere. Is there a way to let the user see a message to the effect of "I'm working on it" and then see the result as soon as it's done? Or is this too dynamic? Is there no way to be free of reliance on cron or some other cleanup entity? Must I actually write out a file?