Vautrin has asked for the wisdom of the Perl Monks concerning the following question:

I have a CGI script that I tested very thouroughly and ran from the console before taking the module and putting it into a CGI script. The problem is that although the script ran quite well from the console, I ended up getting this error every time I ran it over the web, after it would die in the middle of running for a very long time:

(70007)The timeout specified has expired:
ap_content_length_filter: apr_bucket_read() failed

[download]

So I ran a google search for that error and didn't come up with everything. I realized that the script takes a pretty long time to execute, so I tried setting $| = 1 at the beginning of my script. This lets the script run about half way through (the output scrolls down the page) until it eventually dies.

So I went through and changed some things (I reduced redundant data processing and sped things up a little) and it went even farther through (3/4s of the way), but still won't complete.

Does anyone know how I can tell apache (which I assume is killing the process because it thinks it is hung) to let it finish? It's killing me that this program gets to within completion but can't complete. I'm running Apache 2.0.47 without mod_perl. The best idea I've had so far is to fork the process, and have the page update itself after 60 seconds or so, but this seems like too much of a pain.

Update 1: I installed mod_perl and am still reading the documentation, but it doesn't seem to do anything to speed up the script. (i.e. its dying at the same place it died before). Not sure if this is because I installed from RPMs though, still checking.

Update 2: Thank you everyone who responded. I ended up using forks and javascript and a directory to store the files in order to finish off the problem. I don't have any extra votes right now, but for everyone who helped I am going to ++ you tomorrow.

For anybody searching the archives for similar posts, the code that I finally used to get around the problem is below:


use strict;
use warnings;
use CGI;

if ($CGI->param('check')) {
  my $filename = "./xml/" . $CGI->param('username');
  if (-e "$filename.finished") {
    open ("XML", "< $filename")
      or die ("Can't open the file $filename");
    print $CGI->header({-type => 'application/xml'});
    while (<XML>) {
      print $_;
    }
    close ("XML");
    unlink ($filename);
    unlink ("$filename.finished");
    exit;
  }
  else {
    print $CGI->header({-type => 'text/html'});
    print $CGI->start_html;
    print "<meta http-equiv=\"refresh\" content=\"3;url=\"foo.cgi?chec
+k=1&username=". $CGI->param('username') . "\">\n";
    print $CGI->h4("User ", $CGI->param('username')), "\n";
    print $CGI->h4("Password ", "******"), "\n";
    print $CGI->p("The script is now generating your XML report in the
+ background.  The page will reload when it is ready.");
    print $CGI->h4("Unfortunately the script is not yet done.");
    exit;
  }
  exit;
}
elsif ($CGI->param('do')) {
  my $pid = fork;
  if (not (defined $pid)) {
    die ("The \$pid was not defined");
  }
  elsif ($pid) {
    print $CGI->header({-type => 'text/html'});
    print $CGI->start_html;
    print "<meta http-equiv=\"refresh\" content=\"3;url=\"foo.cgi?chec
+k=1&username=". $CGI->param('username') . "\">\n";
    print $CGI->h4("User ", $CGI->param('username')), "\n";
    print $CGI->h4("Password ", "******"), "\n";
    print $CGI->p("The script is now generating your XML report in the
+ background.  The page will reload when it is ready.");
    print "<script>document.location = \"scraper.cgi?check=1&username=
+" . $CGI->param('username') . "\"</script>\n";
    exit;
  }
  else {
    main::generate_report;
  }
}

[download]

I welcome any comments as to how this script could be improved.

Update 3: The above code had a few errors from cut and pasting without context that I fixed.

Thanks in advance,

Vautrin

janitored by ybiC: Balanced <readmore> tags around long code block

Replies are listed 'Best First'.
Re: Need help with CGI script that won't complete running
by bart (Canon) on Jan 26, 2004 at 20:05 UTC
    You'll have to launch a background process that does the actual job, and let the CGI script return quickly. That way you'll prevent people that aren't patient from interrupting the script, by pressing cancel on their browser. For basic ideas on how to do it, check out merlyn's gem series, the columns for the now defunct magazine WebTechiques, on his website.

    In particular, check out column 20 of December 1997: Search in progress page.

    My idea is for the script to fork, the parent to quickly finish; maybe let the child fork and exit again, so the parent script can wait for it; and letting the grandchild do the actual work.

    Do check out some docs on zombie prevention.

[reply]
Re: Need help with CGI script that won't complete running
by CountZero (Bishop) on Jan 26, 2004 at 19:44 UTC
    It is quite normal that you do not see a speed-up by using mod-perl. Mod-perl "compiles" your script the first time it is run (so the first time it is run it takes as long as a "normal" perl-script), and as from the second time it is run it uses the pre-compiled version, thus saving you the time to start a new perl-interpreter, loading in the program code and all modules needed, compiling the script and finally running it.

    As you never even get around to run the script once and Apache kills it, mod-perl never gets a chance to show its usefulness.

    CountZero

    "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[reply]

      I see. So I need to get the script running before mod_perl will be able to help? I don't suppose there is a way to compile a version running from the console, is there?

[reply]
        Indeed, you will only see the speed-up the second time the script runs on that interpreter.

        As a rule your scripts should run under standard CGI before you try mod-perl as there are already enough pitfalls when using mod-perl, that you do not need the errors in your script on top of that.

        As a matter of fact, when you run your script from the console it does get compiled before it is run: but when the script ends Perl wipes it memory so next time you start afresh. Only mod-perl is able to "remember" the compiled script.

        CountZero

        "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[reply]
Re: Need help with CGI script that won't complete running
by blue_cowdawg (Monsignor) on Jan 26, 2004 at 19:01 UTC

    Have you tried checking the Apache logs? I hace never had trouble with Apache killing a process. Quite the opposite I've had processes run away from Apache and load down the system.

    Peter L. Berghold -- Unix Professional
    Peter at Berghold dot Net
       Dog trainer, dog agility exhibitor, brewer of fine Belgian style ales. Happiness is a warm, tired, contented dog curled up at your side and a good Belgian ale in your chalice.
[reply]

      The line I quoted in <code> tags was from the Apache Error log. So, that's what Apache is saying about the script. Of course, whether it was perl printing to STDERR that was redirected to the Apache error logs or Apache, I don't know.

[reply]

        Without code to look at I'm going to take a shot in the dark and hope that I hit something without causing a "blue on blue" incident.

        Try using CGI::Carp such that 

        use CGI::Carp qw/ fatalsToBrowser/; #remove from production code!

        [download]
        and see what gets written to your browser. Also do you have any code that could be going into an infinite looop?

        Just a couple of ideas.

        Peter L. Berghold -- Unix Professional
        Peter at Berghold dot Net
           Dog trainer, dog agility exhibitor, brewer of fine Belgian style ales. Happiness is a warm, tired, contented dog curled up at your side and a good Belgian ale in your chalice.
[reply]
[d/l]
Re: Need help with CGI script that won't complete running
by jdtoronto (Prior) on Jan 26, 2004 at 20:22 UTC
    This error seems to be routinely reported since about September 2003. I can only see it being associated with Apache 2 and with scripts that take a long time to run.

    If it runs okay on the command line then I would suspect thenext step is to try it on an Apache v1 server and see what happens then. There is some question about a possible problem in the http protocol code.

    jdtoronto

    Updated Corrected year in para1, thanks Nkuvu

[reply]
Re: Need help with CGI script that won't complete running
by shotgunefx (Parson) on Jan 27, 2004 at 02:30 UTC
    I've got the same problem migrating some clients apps to apache 2. The script which under apache 1.3.x runs in 2 minutes, times out after 12 on the new machine. The new machine is about 4x faster than the old. I believe in my case, it's a bug with apache and it's new filter system. I made it a command line utility and it runs in about a minute. So I've isolated everything else (the filesystem,db, process size) It doesn't fork anything BTW.

    I've read various things that indicate under certain circumstances, that Apache 2 reports prematurarly, the wrong content length because of the way it chunks output for the filters. So first, try speeding it up or increasing the timeout parameter in Apache. Second, Pray. Every report of this I saw while researching, none had an answer (Google with the timeout error text ). (Redhat hasn't been able to come up with anything for me.)

    If I find anything out, I'll be sure to post an answer and would appreciate it, if you could do the same.


    -Lee

    "To be civilized is to deny one's nature."
[reply]

      The interesting thing about this whole thing is that once I forked it, and the process was able to complete (and mod_perl successfully cached the forked compilation) the script flew (10 seconds versus several minutes). This is important to note because originally I tried installing mod_perl and it actually hurt the performance of the script because not only was it trying to run my script but it was trying to cache the compilation (and thus added overhead). This makes me think it should be possible to run the script in the background, and create a web page to "jump" from one web page to the other -- i.e. through refresh or Javascript.

      If you could make the transition nice enough I don't think the users would notice -- especially on the sped up version. Oh, one more thing, the reason I had to assign document.location = newURL; in javascript is because Mozilla choked on the refresh tags. I think it was caching it but didn't fiddle around with it enough to find out.

[reply]
[d/l]
Re: Need help with CGI script that won't complete running
by schumi (Hermit) on Jan 27, 2004 at 15:16 UTC
    I've seen the same error on some of my scripts after migrating from Apache 1.* to Apache 2, and as those mentioning it before me, have ruled out any other reasons for failure. It seems indeed something to do with Apache 2's new filter system. So far the only solution I've come up with is rewriting the scripts (I wasn't very proud of that code anyway...). If someone finds another solution I'd be more than happy to hear it.

    --cs

    There are nights when the wolves are silent and only the moon howls. - George Carlin

[reply]
Re: Need help with CGI script that won't complete (FIX?)
by shotgunefx (Parson) on Jan 27, 2004 at 22:42 UTC
    I've come up with a solution that solved my problem. I made a filter that unsets the content length. I don't have Apache 2 installed here, I wrote this blind, walked the client through installing and configuring it and SWOOSH, fixed the problem on the first try. (Love that feeling.. so very rare.)

    It's got a little bit of kruft (see $context) because I based it on a module featured on perl.com about Apache2 Filters. It will probably work if you remove the $context code but I'm not sure and don't have apache 2 to test.. (Can we get at the filters from mod_cgi in 2.x?, do we have to do NPH? It would be nice if we could just fix this in CGI with a use statement instead of screwing with Apache's config...)

    I'm going to try and get 2.* running on a box here so I can remove the kruft.

    Update
    Removed kruft and tested.
    Read more... (3 kB)
[reply]
[d/l]
Re: Need help with CGI script that won't complete running
by dragonchild (Archbishop) on Jan 29, 2004 at 12:35 UTC
    I've been following this thread with some dismay. I'm running Apache2 and I've got scripts that can take up to 5 minutes to complete. But, they always complete. This is on both mod_cgi/RH9 and mod_perl/Solaris9. There's got to be something else involved.

    ------
    We are the carpenters and bricklayers of the Information Age.

    Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

[reply]
      I've recently come across the same problem migrating to Apache 2.x.

      I found some useful information here and here. Looks to be more of a stderr problem in mod_cgi. Some of the proposed hacks have helped or worked.

      Please help post any more help anyone can find. This is an important problem to resolve.

      peppiv

[reply]

Back to Seekers of Perl Wisdom