Frustrating LWP timeouts -- help!

sghani has asked for the wisdom of the Perl Monks concerning the following question:

I'm running into an extremely frustrating situation where I'm trying to POST the contents of a very large file, but appear to be running into timeout issues. I'm working with a Google Search Appliance where the only way to get data into it is through a standard HTTP POST (other than having it crawl, of course). Here's the snippet where the actual POST is taking place:

    # Post content to the supplied URL
    my $ua = new LWP::UserAgent;
    $ua->timeout( 3600 );
    my $response = $ua->post(
        $url,
        Content_Type => 'multipart/form-data',
        Content => [ 
            feedtype => $feedtype,
            datasource => $datasource,
            data => [
                $xmlfile,
                $filename,
                Content_Type => $mimetype
            ]
        ]
    );

    # Final error check
    throw Error::Simple( $response->message ) if( !$response->is_succe
+ss );
    print "Success";
}
catch Error::Simple with {
    my $E = shift;
    print "ERROR: ", $E->text;
}
finally {
    print "\n";
};
[download]

In the above, $xmlfile is the actual location of the XML file being posted, while $filename is the name of the XML file itself. The file I'm trying to post is about 100MB (and sometimes even larger). Unfortunately, after a few minutes of running the script, I get the following error:

 server closed connection without any data back
[download]

At other times, I'm simply getting an EOF and then sudden death. I've tried everything known to me without any luck at all. Thinking that it may be a timeout issue, I set it to an hour. That didn't appear to solve the issue at all. Any help here would be enormously appreciated. Thanks, Saker

Comment on Frustrating LWP timeouts -- help! Select or Download Code

Replies are listed 'Best First'.
Re: Frustrating LWP timeouts -- help! by Khen1950fx (Canon) on Aug 26, 2006 at 07:50 UTC
As I see it, there are a couple of things that you can do here. The first thing would be to alter the timeout in the script: `#Post content to the supplied URL my $ua = LWP::UserAgent->new; $ua->timeout ( 0 );` [download] The 0 time out is just a starting point. You can use any value that you want, but I would keep that value smaller than 3600. If that doesn't work, then go into LWP::UserAgent with your editor and comment out the timeout statements.	[reply] [d/l]
Re^2: Frustrating LWP timeouts -- help! by sghani (Initiate) on Aug 26, 2006 at 16:20 UTC
I tried your suggestion; unfortunately, I'm getting the same results -- `server closed connection without sending any data back` [download] What's somewhat odd (not entirely, I guess) is that the above snippet works perfectly when trying to push files of a smaller size, e.g. 50MB -- it's only happening with larger one. I will be trying the commenting out idea suggested by you. Is there anything else to try if that fails too? Thanks for all your help -- really appreciate it.	[reply] [d/l]
Re^3: Frustrating LWP timeouts -- help! by Khen1950fx (Canon) on Aug 26, 2006 at 17:42 UTC
Thank you for the reply. I'm fairly certain that "commenting out" will work, but if it doesn't, then try LWP::UserAgent::Determined. Use $browser->timing("1,3,15"). LWP::UserAgent::Determined	[reply]
Re: Frustrating LWP timeouts -- help! by bobf (Monsignor) on Aug 26, 2006 at 18:27 UTC
If increasing the timeout value doesn't work, maybe timing out isn't the problem. Does the server limit the amount of data that can be uploaded through a POST? You mentioned that 50 MB will work - you could try increasing that size until you start getting failures, and if they occur consistently around a given POST size that might be a clue.	[reply]
Re: Frustrating LWP timeouts -- help! by Anonymous Monk on Aug 26, 2006 at 11:17 UTC
I've tried everything known to me without any luck at all. What is that? LWP::Debug?	[reply]
Re: Frustrating LWP timeouts -- help! by mda2 (Hermit) on Aug 28, 2006 at 03:28 UTC
I think it isn´t an error on client side... is a server side limit, or other protection for post size! Are you using a web server ? If yes, try to see error log... For this data size, try to split in more sequential posts... -- Marco Antonio Rio-PM	[reply]