Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm uploading files in my CGI::Application app, some of them are going to be ~4GB. I think I'm suffering from buffering or something. At the moment a 3.5GB file is failing to upload at around th 10% mark. I've checked the disk space and it's ok. I've added some debug lines to warn out some values so that I can get an idea of what's going on, but none of it is displayed until the file upload has completed. It works well for files we've tested so far, just not this large one.
sub Upload{ my $self = shift; my $q = $self->query(); my $emailbody; my $bytes_retreived = 0; my $bufsize = 1024; my $buffer = ''; my $fh = $q->upload('upfile'); my $filename = uc( $q->param('upfile') ); return '' if ! $filename; my $tmp = File::Temp->new( DIR => $self->cfg('UploadDIR'), SUFFIX => '.zip', UNLINK => 0, ); warn "temp file - $tmp"; binmode $fh; while (read ($fh, $buffer, $bufsize)) { print $tmp $buffer; $bytes_retreived += $bufsize; warn "bytes recieved: $bytes_retreived"; # I expect to see thi +s in real time when tailing the error_log } close $fh; #close $tmp; my $file = $self->session->param('UPLOADPATH') . "/" . $filename; warn "file -- $file"; move($tmp, $file) or die "file move failed $!"; # some more code follows which isn't relevant to the problem. }

Replies are listed 'Best First'.
Re: CGI::Application file uploads buffering
by cjb (Friar) on Mar 24, 2011 at 14:11 UTC

    What is $| set to?

      thanks, it's set to $|++; near the top of my script. Do I need to put this somewhere special for CGI::Application?
      ALso I note that there's a temp file in /var/tmp called CGItempXXXX. Why is this here if I'm using file::temp?

        perldoc seems to indicate "under Accessing the temp files directly" that's expected behavior for CGI.pm. You might want to look at the section "Progress bars for file uploads and avoiding temp files"

Re: CGI::Application file uploads buffering
by locked_user sundialsvc4 (Abbot) on Mar 24, 2011 at 17:11 UTC

    I shudder to contemplate the transfer of a 4GB file, in one chunk, using the HTTP protocol ...   The number of bytes transferred will be quite a bit larger than that, due to encoding.

      Why? This happens all the time, people download DVD images over HTTP.
        Why? This happens all the time, people download DVD images over HTTP.

        We're talking about uploading, and since Content-Transfer-Encoding could be Base64, the number of bytes could be double.

Re: CGI::Application file uploads buffering
by Anonymous Monk on Mar 24, 2011 at 14:02 UTC
    More info: When testing smaller files right now I don't see tel file being written to on the system, I don't see the file write till after it's all uploaded.
Re: CGI::Application file uploads buffering
by Anonymous Monk on Mar 30, 2011 at 15:01 UTC
    Bump, does anyone else have any ideas?
      Study your copy of CGI.pm very very closely, check POST_MAX, etc etc