HTTP::Request GET - url length restriction?

lovelost has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, is there a maximum string length on a GET url when doing a HTTP::Request? I am forced to call a script (which returns a dynamic image) over the net using GET, and my query is unfeasibly long (normally way over 1024 characters). If I try the following:

$request = HTTP::Request->new('GET', $reallylongurl);
$request->content_type('image/gif');
$response = $ua->request($request, '/tmp/blah.gif');
[download]

It then barfs with:

Error while getting [long url here] -- 500 short write
[download]

This error originates from Protocol/http.pm, line 206 (LWP version 5.69 I believe). Entering the URL in the shell via wget, or even pasting it into Mozilla works fine (the server at the other end does not pose length restrictions). The above also works fine when it just so happens that my query is fairly short, although I have not tested precisely at what length this problem occurs, since someone may have come across this and knows an easy way around! Apologies if I am making a stupid error as I've not much experience with the LWP and associated routines.

Many thanks.

Comment on HTTP::Request GET - url length restriction? Select or Download Code

Replies are listed 'Best First'.
Re: HTTP::Request GET - url length restriction? by William G. Davis (Friar) on Jan 12, 2004 at 19:55 UTC
That error means LWP syswrite()'ed your very, very long request and when syswrite() returned the number of bytes it wrote, it ended up being less than the what LWP told it to write. Could you please post a request that causes this problem? Also, what's your OS and what's your Perl version?	[reply]
Re: HTTP::Request GET - url length restriction? by hardburn (Abbot) on Jan 12, 2004 at 21:17 UTC
There is a maximum size a server will hold for a GET request. IIRC, it's usually 1024 bytes, but this may change from server to server. If you need more than that, use a POST. In general, you should probably be using POST whenever possible, unless you need to be proxy-freindly. ---- I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident. -- Schemer `: () { :\|:& };:` Note: All code is untested, unless otherwise stated	[reply] [d/l]
Re: Re: HTTP::Request GET - url length restriction? by lovelost (Initiate) on Jan 13, 2004 at 09:33 UTC
To be more specific, basically I am calling a mapping server with lots of postcodes, which then returns a map with those postcodes represented on the map. The company can only accept GET requests. Their server accepts very long urls, because my very long test example works fine with wget from the shell, and even if the GET request is posted as a URL in Mozilla / Netscrape. I can't give an example because it would break data protection law, but the get requests will be bigger than 1K. Note that the request will work with LWP if the request is a lot smaller. I'm restricted to using Perl 5.6.1 on RedHat 8. LWP is 5.6.9 I believe. Cheers	[reply]
Re: Re: Re: HTTP::Request GET - url length restriction? by William G. Davis (Friar) on Jan 13, 2004 at 12:24 UTC
Well, if you've tried the EXACT same request using Mozilla, Netscape, and wget and it still only broke under LWP, then--just out of curiosity--how do you know it worked under Mozilla, Netscape, or wget? Are you sure the server received the entire request with them, or did they just not raise errors like LWP did? You might want to contact Gisle Aas about this via the CPAN request tracker.	[reply]