Madcap Monks,

I often write small scripts to do simple processing to large files (~1Gb and ~18-million lines). These scripts are launched from a link on an admin web page, so I usually include a print statement to show progress.

Running the script below, I expected to see ", $respondent" fill my screen repeatedly as the script went through the file. Instead, the browser just sat there until it timed out and gave me its default timeout message "Server Error". I know, by looking at the files in questin, that the script is still running and doing its job.

Why do the prints not print in real time?

Thanks.

    open(FILE, $data_file) or die("cannot open file :  $!");
    print "<p>opened file $data_file ok\n";
    $a = 0;
    while ($line = <FILE>) {
        $a++;
        #record every thousand lines as way to check progress
        if ($a % 1000 == 0) {
            open(pFILE, '>>progress.txt') or die("cannot open progress
+ file : $!");
            print pFILE ", $a";
            close(pFILE);
        }
        #parse the line
        @line = split(//, $line);
        my $line_id = join('', $line[6], $line[7], $line[8]);

        if ($line_id  eq $target_line_id) {
            my $respondent = join('', $line[0], $line[1], $line[2], $l
+ine[3]);
            # if this line meets the criteria
            if (($line[$col_1 - 1] eq $value_1) && ($line[$col_2 - 1] 
+eq $value_2)) {
                $file = ">>".$data_dir."/t".$rid;
                #record this respondent in a text file
                open(rFILE, $file) or dienice("cannot open file : $_[0
+] $!");
                print rFILE "$respondent\n";
                close(rFILE);
                # output progress to browser
                print ", $respondent";
            }
        }
    }
    close(FILE);
[download]