Re^2: Speeding up large file processing

How often does the information you are extracting from the directory structure change?

Seems to me that instead of rebuilding the html representing the structure every time the cgi script is called, you should be maintaining a pre-built file that contains the html.

When a request arrives to display the data, you just read that pre-formatted file and present it.

When changes are made to the directory structure, you run a server process that re-creates the html file.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

Comment on Re^2: Speeding up large file processing

Replies are listed 'Best First'.
Re^3: Speeding up large file processing by ralphch (Sexton) on Jul 15, 2005 at 16:35 UTC
Hi, thanks for your reply! Yeah, I thought of doing something like that to speed it even further. However, the information changes randomly at different times since I'm not the one updating the content. For now, this solution is running at least fast enough for the current load. Regards, Ralph.	[reply]
Re^4: Speeding up large file processing by BrowserUk (Patriarch) on Jul 15, 2005 at 17:11 UTC
In that case, I'd think seriously about setting up a daemon process to monitor the directory structure and update the html when it detects changes. Under Win32 I'd use Win32::ChangeNotify. There is a similar facility (PAM?) under Linux I believe. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.	[reply]