Does the script time out, or does the browser time out waiting for a response from the web server? If the latter, then your script needs to turn off buffering, and arrange to emit some form of periodic "I'm still alive" message to the browser while you're parsing log files. One non-obtrusive way of doing this is to periodically print an HTML comment. This can double as debug info, allowing you to see trace info from your script by viewing source.

Try this: as you begin parsing each new log, do   print "<!-- $logfilename -->\n"; This will help keep the browser from timing out on you.

The following happens after browser has made a connection.

1. The Apache TimeOut directive controls when the webserver will give up and close connection when waiting to recieve data (from browser) or send data (static or cgi generated data). The TimeOut default in my version of Apache is 300 seconds.
2. The human controlling the browser can also kill connection via STOP button or killing browser.

... It works fine when it is a small report that only takes 5 minutes or less, but if it has to parse a large amount of traffic over many log files, the script timesout before it is complete.

Good Luck

mitd-Made in the Dark
'Interactive! Paper tape is interactive!
If you don't believe me I can show you my paper cut scars!'

I believe that it is never a good idea to have user waiting.

Not quite what you meant by "waiting", but merlyn has a column that describes how to give users periodic feedback on long searches.

Yup, right on! If you do have to make'em wait let them know whats going on. My point ( not really well made :) ) was alot /perhaps most of the time these kinda problems don't need realtime processing.

mitd-Made in the Dark
'Interactive! Paper tape is interactive!
If you don't believe me I can show you my paper cut scars!'

mod_perl and CGI.pm aren't mutually exclusive; mod_perl embeds a Perl interpreter in the Apache binary; apart from saving the overhead of starting the interpreter and compiling the Perl script for each time it is requested, it also gives you 'hooks' into the handling of the request. So it's faster and far more powerful than traditional CGI scripts (which run outside of the web server and typically only communicate with it via the environment variables), but by itself it doesn't parse user input or generate HTML. Of course, in a mod_perl environment it makes sense to implement a templating system, so you may not need the latter.

I mistrust all systematizers and avoid them. The will to a system shows a lack of integrity

1. I think its generally the browser that controls the timeout. You can keep it running by continuing to pass data to it (print some characters occasionally. possibly in a comment) the server can deal with it, but unless its all the server does it's probably not a great idea increasing the timeout. if you do it'll affect all the other pages as well.
2. have the cgi-script act as a bootstrap for the actual script that does the work (exec the program. rather than system; if the cgi script is terminated (the browser is closed) the children would be killed.) (i think)
3. Hmm, havn't read much about mod_perl, probably worth using if you use the script a lot, otherwise , i'm not so sure.

If you choose to use mod_perl, you can do a bunch of stuff after the request is served using the cleanup handler. This will definitely allow you to have the effect you want for your point 2

If you have a CGI that takes a long time, I suggest to use Randal's suggestion, which is described here

Just my 0.02$ worth.

---------------------------
Dr. Mark Ceulemans
Senior Consultant
IT Masters, Belgium