yegg has asked for the wisdom of the Perl Monks concerning the following question:

I need help with a "Premature end of script headers:" Apache/CGI problem, but it is not the normal situation where this error occurs. That is, it doesn't happen all the time.

When I start Apache, everything works fine, for about a day or so (thought not exactly a day). After the critical time passes, all attempted regular executions (not under mod_perl) of Perl CGI scripts end with a 500 error and "Premature end of script headers:" in the Apache error log.

I was running 2.2.8 on Windows with ActiveState build 5.8.8 Build 222. I just upgraded to 2.2.9 in hope it will fix this problem, though I doubt it.

I've looked at permissions and various httpd.conf settings including ScriptInterpreterSource (and messing with the shebang line). I've tried using Carp, but no errors get printed to the browser and nothing more gets printed in the Apache error log with LogLevel Debug. Also, nothing is in the Event Viewer (although I wouldn't expect anything).

Another interesting wrinkle is that when the switch happens and regular CGI scripts fail across the board, CGI scripts run under mod_perl (even the same scripts) continue to work just fine.

Of course, restarting the server corrects the problem (until it happens again in another day or so).

Oh, and I've also tried load testing CGI scripts to see if it reaches a certain threshold # of requests it will kick over. This didn't work either.

Any ideas would obviouslly be appreciated. I have a 10 min monitor on a script now so I'll know when it goes down again.

  • Comment on Not your normal " Premature end of script headers:"

Replies are listed 'Best First'.
Re: Not your normal " Premature end of script headers:"
by RMGir (Prior) on Jun 18, 2008 at 14:01 UTC
    Hmmm, are there any quota issues involved? Those often look quite bizarre from the outside.

    Maybe one of your CGI's isn't exiting correctly, so the httpd has too many child pids?

    Or there's a filehandle leak somewhere and the later child processes can't open stdout?

    Wild guesses all, but an lsof on the httpd and a ps checking on its child processes might point you to a better guess.

    If that fails, truss or strace the httpd process with the "follow children" option once this starts happening, and see where the failure is coming from.


    Mike
      If that fails, truss or strace ...

      As the OP seems to be on Windows, StraceNT might be the tool of choice here... (but I'd second the suggestion in general, of course).   Also, it's probably a good idea to run a few tests outside of the webserver environment, to get familiar with the tool first...

        Downloaded StraceNT, and took a trace within that state, but I'm not really sure what I'm looking for. Any pointers?
      OK, it happened again.

      In that state:

      httpd ~330MB Threads: 500 User Objects: 423 Handles: 2,530 GDI Objects: 4 Context Switches: 28,475

      On reload:

      ~28-50MB pretty quick and growing Threads: 502 User Objects: 3-15 pretty quick Handles: 902 GDI Objects: 4 Context switches: 325-3000
      I got this using TaskManager and ProcessExplorer. I also took a trace using StraceNT, but it doesn't mean anything to me.

      Does anyone know where limits would be set? I have plenty of memory 4GB, and a postgresql process can go up to 756MB.

      When the server gets in this state, it happens for all non-mod_perl CGIs. I also made the simplest CGI I could think of for testing, which also fails:
      #!C:/Perl/bin/perl use strict; print "Status: 200 OK Test";
      If I then put this same script under mod_perl, it works fine. I will do what you said and report back with any findings or non-findings.

      I am getting the sense that the script isn't being executed at all.

Re: Not your normal " Premature end of script headers:"
by hangon (Deacon) on Jun 18, 2008 at 17:55 UTC

    Have you tried parsing your access log for clues? Looking for things such as a particular script that runs right before the problem occurs, excessively long/suspicious GET requests, and whatever else comes to mind may help point you in the right direction. This may take a little grunt work, but you can whip up some Perl to help you out.

    You could also temporarily configure additional logs to help isolate the problem. Besides Apache's normal log formatting parameters, you can use a subset of Perl type regexes for filtering. Here's an example Apache config (untested) for only logging requests to cgi scripts.

    SetEnvIf Request_URI "/cgi-bin/.+\.cgi" logthis=1 CustomLog /path-to/access_log_temp combined env=logthis
      That's a good idea. Another option: count how often each script is run before the first error.

      Perhaps you'll be lucky and script x will be run exactly the same # of times before the failure each restart...


      Mike
Re: Not your normal " Premature end of script headers:"
by Anonymous Monk on Jun 18, 2008 at 13:15 UTC
    only perl cgi, or php cgi also? I think its best to contact apache support forum
      Not sure. I don't have PHP installed.
Re: Not your normal " Premature end of script headers:"
by mr_mischief (Monsignor) on Jun 18, 2008 at 16:32 UTC
    When you say "Carp", do you mean Carp or do you mean CGI::Carp? Since this is CGI, you may want:

    use CGI::Carp qw( fatalsToBrowser );

    This gives you the error through the server that ends up in your browser, if it can, instead of through your log files. It's a nice, quick tool to use when debugging this sort of thing. It takes care of putting out headers for itself and everything.

      I meant CGI::Carp, and when I did what you suggest, no more additional error information printed out. Does this tell you anything? I would guess it indicates the script is never even executed.
        It could mean it's never executed. That could be due to path issues with the perl program (unlikely, since it works up to that point). It could be due to child program limits or the web server user's process limit. It could be due to a memory limit on the web server process and its children or the web server user. It could mean something, somewhere, is causing a syntax error (unlikely, since it works initially).

        If it happens around the same time of day, it could be due to time range issues in your ACLs (which is also unlikely, since you say restarting the server works no matter when it happens).

        It could mean the CGI program is running but unable to open a file handle for its STDIN to get info from the server or STDOUT to get information back to the server due to file handle limits.

        It's a good idea to know about how many file handles the web server user has open, how much memory all the processes it runs are using, and how many processes it has running when this happens. Then, compare that to the mechanism which limits those things. It's ulimit on Unixy systems; I don't remember the name of the policy mechanism that handles such things on Windows.

Re: Not your normal " Premature end of script headers:"
by CountZero (Bishop) on Jun 18, 2008 at 21:10 UTC
    By definition, in order to get this server-generated error, your script(s) must generate some output before the server (or in some cases the scripts) had the opportunity to output and finish the required headers.

    As it concerns all your CGI scripts, it must be something not linked to any single individual script, but must be more general.

    Therefore your first task should be to hunt for the exact output made by your failing scripts. Most likely it is an error message and that normally goes into the Apache error log and CGI::Carp should be able to catch it.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

      While the problem is more general, the nature of the problem suggests that it is triggered by an event, or an accumulation or sequence of events. A script may or may not be triggering the problem, but determining or eliminating this possibility is a step toward resolution.

      Could a particular execution path in a script change something in the environment? Could it trigger a bug in Apache? Could a bug in an XS module overwrite some memory location that it shouldn't? Could the problem be related to POST or GET parameters sent to a script? Maybe.

      To be more accurate, it means the process Apache is calling produces an insufficient set of headers. There need not be any output at all, let alone at any specific point in time relative to something else. If there is no content-type header, the error is issued.

      The lack of output or incorrect output could be caused by many things. It could be caused by STDOUT being redirected elsewhere. It could be an error condition in the shell or the script interpreter/compiler is the only output. It could mean the script runs but produces no output at all. It could mean the output is in the wrong order, or that it doesn't include the content-type header at all.

      The reason for the error is often found in the error log, such as

      [Mon Jun 23 10:05:39 2008] [error] [client 192.168.1.253] (13)Permissi +on denied: exec of '/var/www/vhosts/chris.test/foo.pl' failed
      which would then be followed in the stock error log format by another entry like
      [Mon Jun 23 10:05:39 2008] [error] [client 192.168.1.253] Premature en +d of script headers: foo.pl

      If the script simply produces no output at all or produces output which doesn't include the content type header, you'll just get the one error logged about premature end of script headers.