tl;dr: Somehow a CGI script that doesn't write to disk kept running for about 16 hours after the client disconnected, filled up the disk about 10 hours in, and then freed the space when Apache was killed. Contents of script unknown.

Fully patched CentOS 7. Woke up this morning to "Disk quota exceeded" errors and: 

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/simfs 25G 25G 0 100% /
# du -sh
3.9G .

[download]
Top indicated that I had plenty of ram left and a CGI script I wrote yesterday was the likely culprit: 
KiB Mem:   1048576 total,   380264 used,   668312 free,        0 buffe
+rs
KiB Swap:   262144 total,    81204 used,   180940 free.    33856 cache
+d Mem
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COM
+MAND
 5140 apache    20   0  239888   2348   1756 R 17.3  0.2 144:42.67 htt
+pd
14980 apache    20   0   30840   1884   1228 S 15.6  0.2 153:43.94 bou
+nced.cgi

[download]
I killed Apache and now my disk and cpu utilization are normal. I didn't have lsof installed so I couldn't see what file was causing the problem.

Access_log shows only me accessing the script, and error_log shows nothing since I wrote it.

I wrote this quickly yesterday with no error handling, but worst I expected to happen if there was an error was the script to die. I can't understand how the following could possibly fill up my disk. It appears to work as intended. 

#!/usr/bin/perl

use strict;
use warnings;
use CGI;
use CGI::Carp qw(fatalsToBrowser);

my $q = new CGI;

print $q->header;
print $q->start_html('Bounce summary');

my @files = </home/user/Maildir/cur/*>;
for (@files){
    open(IN, $_);
    while (<IN>) {
        last if /The mail system/;
    }
    while (<IN>) {
        if (m#/domain.com$#) {
            print '<p>';
            last;
        }
        s/</&lt/g;
        print;
    }
    close IN;
}

print $q->end_html;

[download]
Edited to add:
Pulling the CGI components out gives nearly identical output to what the web browser tab displays, with no errors showing. The directories I was/will run this against never have subdirectories.

Having thought about it today, I believe one of my initial assumptions when opening this thread was probably incorrect. As a CGI script it only runs when I access it. I only ran it a couple times in its final state (above). It is probable the stuck version was different, and I simply didn't notice it. It could have run for many hours before crippling the server. I do not make a habit of confirming scripts end when they stop loading or I hit X in my web browser, I just assume Apache will kill them.

I just really don't understand how a cgi script could stay running without a client attached. I just created one with an intentional infinite loop, and as soon as I hit X Apache killed it.

From /var/log/messages after I ran "service httpd stop" this morning: 

Nov 19 10:38:44 systemd: httpd.service stopping timed out. Killing.
Nov 19 10:38:44 systemd: httpd.service: main process exited, code=kill
+ed, status=9/KILL
Nov 19 10:38:44 systemd: Unit httpd.service entered failed state.

[download]
"kill -9 14980" probably would have fixed the problem without killing Apache, but I didn't think of it at the time.

Update 2:
It is actually trivial to create a cgi script that won't die when the client disconnects. My test above contained a "print" inside the loop. Looks like Apache disconnects STDOUT when the client disconnects which causes print to kill the script. For example, a cgi containing just: 

#!/usr/bin/perl
sleep 1 while 1;

[download]
will keep running after the client disconnects, and a "service httpd stop" will yield the same errors as above, however, Apache will kill it after the cgi timeout. So apparently one of my interim scripts entered an infinite loop without a print, but with something that caused Apache's timeout not to kill it. Still no idea how that could use up all free disk space, and then free it immediately when killed.

I just tried writing to STDERR in the loop, both with "print STDERR" and by trying to read a closed filehandle. In both cases error_log recorded the errors immediately and continued to grow in size. When I experienced the disk full error yesterday one of the first things I checked was the log files. error_log was only 7728 bytes.

In reply to Runaway CGI script by Pascal666

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.