LWP config

fourmi has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I'm trying to make

sub GetURL
{
  my ($inURL) = @_;
  my ($ua, $request, $response);
  $ua = new LWP::UserAgent;
  $ua->agent('Mozilla/4.0 (compatible; MSIE 5.22;Mac_PowerPC)');
  $request = new HTTP::Request 'GET', $inURL;
  $response = $ua->request($request);
  if ($response->is_success) {
    return $response->content;
  } else {
    return $response->error_as_HTML;
  }
}
[download]

Emulate a request from my machine as much as possibe, I have already sorted the user agent side, but the other are causing confusion, Ideally i would like to set the following

HTTP_ACCEPT = */*
HTTP_ACCEPT_LANGUAGE = en
HTTP_CONNECTION = Keep-Alive
HTTP_EXTENSION = Security/Remote-Passphrase
HTTP_UA_CPU = PPC
HTTP_UA_OS = MacOS
HTTP_TE = deflate,gzip;q=0.3
[download]

does any know how i go about setting these? The HTTP_TE seems to be set by default, so it is already set, but how would i modify it all the same?
cheers
ant

Comment on LWP config Select or Download Code

Replies are listed 'Best First'.
Re: LWP config by Corion (Patriarch) on Mar 12, 2004 at 12:29 UTC
What you want to set are the headers of the request you send, as that is what the server then sees. The following untested code should set the headers of the request: `$request = HTTP::Request->new('GET', $inURL); $request->header( 'Accept-Content' => '/' ); $request->header( 'Accept-Language' => 'en' ); $request->header( 'Connection' => 'keep-alive' ); $request->header( 'Extension' => 'Security/Remote-Passphrase' ); $request->header( 'UserAgent-CPU' => 'PPC' ); $request->header( 'UserAgent-OS' => 'MacOS' ); $request->header( 'Transfer-Encoding' => 'deflate,gzip;q=0.3' );` [download] This should be the correct headers, but I guessed on many of them, so some more experimentation is in order. Also, you're specifying a `Connection: Keep-alive` header, which won't work properly with the standard uses of LWP, as LWP dosen't support keep-alive connections unless you also make use of LWP::ConnCache. So after some initial experimentation, you might want to change that header to `Connection: close`, which shouldn't affect the rest of the operation. While browsing the documentation of LWP and LWP::ConnCache, it seems that LWP tries to Do The Right Thing if you simply specify a `Connection: keep-alive` header, so it might Just Work™	[reply] [d/l]
Re: Re: LWP config by fourmi (Scribe) on Mar 12, 2004 at 12:41 UTC
I am repeatedly amazed at the speedyness of response here. Thankyou very much, I've been wandering around google, and cpan sites for most of the morning, and now the examples, and links have given me the directions. Much Much Much appreciated!! cheers ant	[reply]