I'm trying to use LWP::UserAgent to automate retrieval of a switch database on a 3Com 3300 switch.

I can view the switch database via the HTTP interface with IE. I can even telnet to the device on port 80 and provide the appropriate HTTP directives to retrieve the HTML representation of the database. The transfer is slow, but appears to be stable.

However, when I use LWP::UserAgent, I get the following error (via $response->status_line):
"500 Line too long (limit is 4096)"

A packet sniffer shows that I get the first 31 or so TCP packets, and then LWP appears to close the TCP connection. A similiar trace of the exchange with IE behind the wheel shows over 140 TCP packets to convey the entire page.

I've tried setting attributes on the $ua object via the max_size() and timeout() methods to no avail. I've attempted to use the ':content_cb' => sub{} key/value pair in the get() method to capture the data, but no data comes in before LWP errors out. I've attempted to use the mirror() method, but I get the same error and no data. I've also tried specifying 'keep_alive' => 1 in the LWP::UserAgent constructor, but I didn't see any change in behavior.

I'm at a loss. Could anyone explain to me why LWP would throw this error, and how I could coerce it to work for me?

Here is the basic framwork of code that I'm using:

#!perl
use strict;
use LWP::UserAgent;
use constant URL => 'http://east/dev01/sd/action_sd.rscr';
my $ua = LWP::UserAgent->new();
$ua->credentials( 'east:80', 'device, '****', '****' );
my $response = $ua->get(URL);
if ($response->is_success) {
    print $response->content;
} else {
    die $response->status_line;
}
[download]

Thanks,
-dpmott


Update: The problem appears to be multi-faceted.
First, the 3Com 3300 series switch is not sending back HTTP headers. This makes it an HTTP 0.9 or prior HTTP server, or worse a non-compliant HTTP server.
Second, the Net::HTTP module (from which LWP::Protocol::http inherits) has a couple of paragraphs for its read_response_headers() method which talk about how it handles non-compliant servers.
The LWP::Protocol::http class passes "'laxed'=>1" to that interface, but that is apparently not by itself enough to manage a response that is > 4k and lacking HTTP response headers. Thus, the solution of setting the MaxLineLength flag in the LWP::Protocol::http::EXTRA_SOCK_OPTS array is actually the correct and (as far as I can tell) only solution to this problem. That flag gets passed into the Net::HTTP constructor and serves to keep the module from dropping an HTTP response due to missing HTTP headers.