I had this same problem a few months ago with a script I built for Warner Bros. The script would load 70,000 surveys, calculate statistical means and medians for each question, then output a master report. The problem was, was that the calculations were SLOW. Sometimes the CGI script took a half hour to run.

The beta testers (heh, myself really) would hit reload after a minute or two, thinking that the script had crashed. Why no response from the server? *click* *click* The server stopped responding completely, it crashed from the concurrent requests (CPU: 14.30). That night I started looking for another job.

Two months of work down the drain.

After I calmed down a bit, I realized that it was okay that the request was slow to execute. That's a lot of surveys we're talking about, users just need a status screen updating them on the status of their request. That's all. No big deal. So I built a minor caching system that cached the reports, and then opened Dreamweaver and built a template. Here's a screenshot of the final status screen in action. (Note the progress bar at the bottom in HTML tables. I think it's cute.)

Now all I had to do was send the browser a redirect to this status page (which has a meta tag requesting itself to be refreshed every 5 seconds), then periodically update it as I continued processing surveys, until I finish, and then I'll redirect it to the master report. Easy as pie. Right?

WRONG!!!

It worked fine in Opera and Netscape but IE wouldn't execute the redirect until AFTER the script finished running!!!! Bastard program. I knew there was a good reason to never trust IE. Evil manipulating bastards, I say. And at this point I was behind schedule, I should have delivered the application days before.

My intuition is that Apache doesn't relinquish the TCP/IP connection (server side) until the perl process exits. That's logical right? IE received the redirect just fine, but it waited for that TCP/IP connection to drop before actually executing it. (There's even a Microsoft knowledge base article on the problem, but I switched computers and don't have it bookmarked any more.)

I'm not a perl guru or anything, so instead of wasting more time figuring out how to tell Apache to drop the connection, and probably learning something in the process, I hacked a crappy fix together that works:

  my $pid;

  #
  # Fork the current process.
  #

  if (!defined ($pid = fork)) { 
      DieNice("Unable to fork: $!\n");
  } elsif (! $pid) { 
      # This is the child fork, for processing.
      close(STDIN); close(STDOUT); close(STDERR);

  } else {
      # This is the parent fork, for the browser.
      print $query->redirect("http://$servername/wbnorm/cache/status.h
+tml");

      # Apache, you will answer to me!
      exit;
  }

  #
  # Calculations continue as normal here
  #
[download]

*ding*!

(I registered after a year of lurking just to participate in this very topic. As a new user, I'd like to say, merlyn -- you've saved my ass dozens of times, and I thank you for it.)