Curling slowly

mungohill has asked for the wisdom of the Perl Monks concerning the following question:

Esteemed Monks,

I am using WWW:Curl::Easy to send a very small update message. Because of the way the service works, I need to do a load of individual performs, in some cases running into the hundreds to send all the required updates. A user is sitting staring at a screen while this is happening and waiting for a result.

Running my client program on, let's call it Machine A, I'm seeing 'verbose' output like this:

* About to connect() to blah.blah.com port 8080
*   Trying xxx.xxx.xxx.xxx... * connected
* Connected to blah.blah.com (xxx.xxx.xxx.xxx) port 8080
> POST /submit.sp HTTP/1.1
Host: blah.blah.com:8080
Accept: */*
Content-Length: 5
Content-Type: application/x-www-form-urlencoded

blah
< HTTP/1.0 200 ITAS Ltd SOAP Server v1.3.53
< Content-length: 343
< Server: ITAS Ltd SOAP Server v1.3.53
< Connection: close
* Closing connection #0
[download]

This works, well, not exactly quickly, but easily fast enough for my purposes. On, let's call it Machine B, I see this:

* About to connect() to blah.blah.com port 8080 (#0)
*   Trying xxx.xxx.xxx.xxx... * connected
* Connected to blah.blah.com (xxx.xxx.xxx.xxx) port 8080 (#0)
> POST /submit.sp HTTP/1.1
Host: blah.blah.com:8080
Accept: */*
Content-Length: 1523
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue

* Done waiting for 100-continue
* HTTP 1.0, assume close after body
< HTTP/1.0 200 ITAS Ltd SOAP Server v1.3.53
< Content-length: 277
< Server: ITAS Ltd SOAP Server v1.3.53
< Connection: close
< 
* Closing connection #0
[download]

On Machine B it takes about a second to execute. You'll notice that there was 'Expect: 100-contine' which wasn't present on Machine A, and indeed, the delay on Machine B is between 'Expect: 100-continue' and '* Done waiting for 100-continue'.

To me, this suggests that, on Machine B, the socket has been created with an SO_KEEPALIVE option. Presumably in response to some default socket options. I know that the keepalive is set up differently on the two machines. Frankly the updates I am doing are not the most crucial things in the world. It is far more important to me that they are done quickly rather than 100% reliably, so I'm looking at ways of speeding it up.

Do you think I'm right in my ramblings about keepalives?

If so, do you think there's any way that I can force curl to create it's socket without keepalives?

Comment on Curling slowly Select or Download Code

Replies are listed 'Best First'.
Re: Curling slowly by Anonymous Monk on Mar 29, 2012 at 10:33 UTC
re Expect100Continue	[reply]
Re^2: Curling slowly by mungohill (Acolyte) on Mar 29, 2012 at 10:46 UTC
That link kind of confirms what I was thinking. Having said that, I can't see any direct way of removing things from the header. I tried completely cleaning out the header by doing this:. `my @myheaders; my $curl = WWW::Curl::Easy->new; $curl->setopt(CURLOPT_HTTPHEADER,\@myheaders); # That should have wiped out the default header assumptions that inc +lude # Expect: 100-continue --which we don't want ### Do the rest of the header stuff via setopt: $curl->setopt(CURLOPT_URL,${uri});` [download] Despite this, it seems to insist on the keepalive stuff	[reply] [d/l]