CGI uploads sometimes fail

Stephen Toney has asked for the wisdom of the Perl Monks concerning the following question:

Dear experts:

We've been using CGI.pm with great success for two years now. However, we recently began to use it for a new application which allows users to fill in a form and include up to 10 images for uploading. We've been having lots of problems whenever there are more than 4 images (and sometimes even with one image).

For example, I uploaded the form with 10 small images (average size 15K) and after 9 minutes I got a response from the server "Bad Gateway". I clicked Refresh to resubmit and the form with all 10 images were accepted in 2.5 minutes. (Even 2.5 minutes seems like a long time for only 150,000 bytes over broadband.)

Other times the server returns the "timeout" message, says the headers are incorrect, or just dies with no message. When the server fails to reply, the temp files are left in the upload area with the last one only partially there.

The failures occur about 1/3 of the time. This pattern has been repeated dozens of times by several testers in different parts of the country, all with broadband. The server failure seems to be worse with more or bigger images, but there is no specific byte-count that triggers it. We have been able to upload a 1.3-MB image when it was the only one attached to the form. $CGI::POST_MAX is set to 5000000.

Current testing is on a brand-new server (Windows Server 2003 and IIS) with no one using it but me. We went to this new server because a shared host at Verio had the same issues, and we thought it was a problem of server load, but that appears not to be the case. The pattern is the same using six different browsers on both Windows and Mac clients. Server connection timeout is set to 15 minutes.

Is anyone aware of any issues with CGI.pm's upload features with regard to multiple images? It seems that a single large image uploads more reliably than numerous small ones.
<br Why would the upload times differ so much between cases?

If I might ask a related question, what is the purpose of uploading to temp files and then moving them, instead of uploading directly to the target location and filename? Seems like an additional load on the server and developer.

Many, many thanks in advance for any advice on this!

Stephen

Comment on CGI uploads sometimes fail

Replies are listed 'Best First'.
Re: CGI uploads sometimes fail by Zaxo (Archbishop) on Sep 25, 2005 at 14:04 UTC
If the problem is in the cgi code, you'll need to show it to us. Can you detect any browser dependence in the problem? It might be a lame client issue. One reason to do the tmpfile dance is to avoid clobbering a current version of the file with an upload which might fail. Server load is unlikely to be a problem. Disk I/O is not fast, but it's much faster than most network connections. After Compline, Zaxo	[reply]
Re^2: CGI uploads sometimes fail by Stephen Toney (Sexton) on Oct 09, 2005 at 01:22 UTC
Zaxo, Thanks for the input; sorry I'm late replying but I've been ill. After much research I've found very few apps that upload more than one image at a time, so we are limiting ours to that also. I really think the conversation between the server and client gets confused; sometimes it works, a few minutes later it doesn't -- with the same files. Thanks again, Stephen	[reply]
Re: CGI uploads sometimes fail by pg (Canon) on Sep 25, 2005 at 15:29 UTC
This could relate to the net speed of the connection. I am not just talking about whether this is a 56k connection or 112k connection. The actual/net speed can relate to some simple facts. I have a good example: Whether the client side was running some "heavy" programs. A client of mine complained that she got timeout from one of our applications. I went to her desk, found that she was listening CD using MS Media Player. I asked her to shut it down, and everything went normal. She was not listening to any online music, it was a local CD, but still the application got less CPU cycles to process all the network communication. Time for a new PC, if she wants to enjoy her music and working at the same time.	[reply]