Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have wrote several programs using the LWP::UserAgent class to send request and parse the information received; however, there seemed to be problems when I try to retrieve the content of certain ASP websites. I use the typical setup: 
  $ua = LWP::UserAgent->new;
  $cookie_jar = HTTP::Cookies->new;
  $ua->cookie_jar($cookie_jar);
  $request = HTTP::Request->new(GET => $url);
  $response = $ua->request($request);

[download]
When I tried to run the program, it just hangs. What I'm suspecting is that ASP is not very friendly toward non-Netscape-IE browsers. The same thing happens when I try to access the website using Lynx on my unix box - it just hangs trying to make connection to the server. One such problem website is "http://www.us.buy.com". Can anyone verify if the same thing happens using Lynx? Does anyone have the same problem and how do you get around this programming wise? Any help is appreciated. Thank you. -mobi

Replies are listed 'Best First'.
Simulate a Specific User Agent with LWP
by Arguile (Hermit) on Oct 16, 2001 at 10:50 UTC

    I was having a similar problem. You might want to look into $ua->agent("string"); in the LWP docs. It allows you to pass yourself off as whatever browser you wish.

    I won't even get into what I think of people who require certain browsers, but unfortunately many do. If they're using server side detection badly, it can thrash the page or completely deny you entry. The answer is to mimic a browser that is allowed.

    The most common restriction is a 4+ version browser, followed by requiring IE only. The following string should solve the problem in most cases:

    Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)

    [download]
    Update: Fixed my horrid horrid late night grammar to just really bad late night grammar. ;)
[reply]
[d/l]
[select]
Re: Simulate Connection using LWP
by blakem (Monsignor) on Oct 16, 2001 at 10:28 UTC
[reply]
[d/l]

Back to Seekers of Perl Wisdom