I have a CGI script that I tested very thouroughly and ran from the console before taking the module and putting it into a CGI script. The problem is that although the script ran quite well from the console, I ended up getting this error every time I ran it over the web, after it would die in the middle of running for a very long time:

(70007)The timeout specified has expired:
ap_content_length_filter: apr_bucket_read() failed
[download]

So I ran a google search for that error and didn't come up with everything. I realized that the script takes a pretty long time to execute, so I tried setting $| = 1 at the beginning of my script. This lets the script run about half way through (the output scrolls down the page) until it eventually dies.

So I went through and changed some things (I reduced redundant data processing and sped things up a little) and it went even farther through (3/4s of the way), but still won't complete.

Does anyone know how I can tell apache (which I assume is killing the process because it thinks it is hung) to let it finish? It's killing me that this program gets to within completion but can't complete. I'm running Apache 2.0.47 without mod_perl. The best idea I've had so far is to fork the process, and have the page update itself after 60 seconds or so, but this seems like too much of a pain.

Update 1: I installed mod_perl and am still reading the documentation, but it doesn't seem to do anything to speed up the script. (i.e. its dying at the same place it died before). Not sure if this is because I installed from RPMs though, still checking.

Update 2: Thank you everyone who responded. I ended up using forks and javascript and a directory to store the files in order to finish off the problem. I don't have any extra votes right now, but for everyone who helped I am going to ++ you tomorrow.

For anybody searching the archives for similar posts, the code that I finally used to get around the problem is below:

use strict;
use warnings;
use CGI;

if ($CGI->param('check')) {
  my $filename = "./xml/" . $CGI->param('username');
  if (-e "$filename.finished") {
    open ("XML", "< $filename")
      or die ("Can't open the file $filename");
    print $CGI->header({-type => 'application/xml'});
    while (<XML>) {
      print $_;
    }
    close ("XML");
    unlink ($filename);
    unlink ("$filename.finished");
    exit;
  }
  else {
    print $CGI->header({-type => 'text/html'});
    print $CGI->start_html;
    print "<meta http-equiv=\"refresh\" content=\"3;url=\"foo.cgi?chec
+k=1&username=". $CGI->param('username') . "\">\n";
    print $CGI->h4("User ", $CGI->param('username')), "\n";
    print $CGI->h4("Password ", "******"), "\n";
    print $CGI->p("The script is now generating your XML report in the
+ background.  The page will reload when it is ready.");
    print $CGI->h4("Unfortunately the script is not yet done.");
    exit;
  }
  exit;
}
elsif ($CGI->param('do')) {
  my $pid = fork;
  if (not (defined $pid)) {
    die ("The \$pid was not defined");
  }
  elsif ($pid) {
    print $CGI->header({-type => 'text/html'});
    print $CGI->start_html;
    print "<meta http-equiv=\"refresh\" content=\"3;url=\"foo.cgi?chec
+k=1&username=". $CGI->param('username') . "\">\n";
    print $CGI->h4("User ", $CGI->param('username')), "\n";
    print $CGI->h4("Password ", "******"), "\n";
    print $CGI->p("The script is now generating your XML report in the
+ background.  The page will reload when it is ready.");
    print "<script>document.location = \"scraper.cgi?check=1&username=
+" . $CGI->param('username') . "\"</script>\n";
    exit;
  }
  else {
    main::generate_report;
  }
}
[download]

I welcome any comments as to how this script could be improved.

Update 3: The above code had a few errors from cut and pasting without context that I fixed.

Thanks in advance,