dallase has asked for the wisdom of the Perl Monks concerning the following question:

the following code results in "500 EOF when chunk header expected" when ran... yet i can pull up http://avantgardeconcierge.com in a web browser and using wget. any ideas what might be going wrong with LWP here? 
  use LWP::UserAgent;

  my $ua = LWP::UserAgent->new;
  $ua->timeout(10);

  my $response = $ua->get('http://avantgardeconcierge.com/');

  if ($response->is_success) {
     print $response->content;  # or whatever
  }
  else {
     die $response->status_line;
  }

[download]

Replies are listed 'Best First'.
Re: 500 EOF when chunk header expected
by marto (Cardinal) on Oct 25, 2005 at 16:15 UTC
    Hmm, http://avantgardeconcierge.com has a frame which opens http://www.golbit.com.
    I get a 504 when trynig to view this via internet explorer (Help they force me to use this!).
    If you are able to view the page in a browser try commenting out the timeout and see what you get.

    Martin
[reply]
      same results with timeout commented. this is a spammer drop page, and i operate http://www.uribl.com, which trie to identify spammer uri's by doing heuristics on their landing pages.... however, i have recently seen an increase in the 500 EOF's, which appears to me like the spammers are doing something sneaky to try to avoid automated detection. i may give WWW::Mechanize a go, and if I have to, revert out to a syscall for wget, but this almost has to be something wrong with LWP dont you think? thanks! d
[reply]
        The URL you have given is http://avantgardeconcierge.com which as I said has a frame which opens http://www.golbit.com.
        The next point in this trail seems to be http://www.tenderise.net seems to do a few things (http://www.tenderise.net/ms/?bb for example, I assme this does some sort of reirect) then fails.

        Im not sure exactly what your trying to achieve, but this does not work from lynx (on my remote netbsb box) or firefox (local), so I wouldnt expect LWP to be any different. If these are your domains why not check the log files to get a better idea of what is going on?

        Hope this helps

        Martin
[reply]
Re: 500 EOF when chunk header expected
by planetscape (Chancellor) on Oct 25, 2005 at 19:20 UTC

    I don't think I can offer much better advice than you've already received, but here goes...

    I am very doubtful that you've discovered a bug in LWP. If it were me, I'd (a) check the server logs, and (b) employ something like Ethereal to help determine what is going on.

    HTH,

    planetscape
[reply]
Re: 500 EOF when chunk header expected
by Anonymous Monk on Sep 09, 2011 at 07:47 UTC
    I have met the same problem, and a solution was to add 
    BEGIN {
$ENV{PERL_LWP_USE_HTTP_10} = 1;
}

    [download]
    or examining LWP/UserAgent.pm 
    if ($ENV{PERL_LWP_USE_HTTP_10}) {
    require LWP::Protocol::http10;
    LWP::Protocol::implementor('http', 'LWP::Protocol::http10');
    eval {
        require LWP::Protocol::https10;
        LWP::Protocol::implementor('https', 'LWP::Protocol::https10');
    };
}

    [download]
    to add the lines enclosed in the if statement directly. Cheers.
[reply]
[d/l]
[select]
Re: 500 EOF when chunk header expected
by Anonymous Monk on Oct 25, 2005 at 17:49 UTC 
     
   use WWW::Mechanize;

    srand;

    my $url = 'http://avantgardeconcierge.com/';
    my $mech = WWW::Mechanize->new();
    my @a = $mech->known_agent_aliases();
    my $agent = $a[int(rand(@a))];
    $mech->agent_alias( $agent );
    $mech->get( $url );

    my $content = $mech->content();

    print "Using Agent: $agent\n";
    print "Web Content: $content\n\n";

    [download]
    results in the same 500 EOF response... probably because WWW::Mechanize->get() wraps the LWP's get() method.

    i just tested a syscall to lynx -source $uri and it works like a champ... but i really wanted to avoid that syscall.

    after looking at the resulting source code produced by lynx, i see dos-style carriage returns following all the headers returned by the avantgardeconcierge.com webserver...

    it says its 'Server: Microsoft-IIS/6.0^M', but all other sites i pull source on using the exact same setup that report 'Server: Microsoft-IIS/6.0' do not have dos-style CR's as EOL markers. maybe LWP is choking on the ^M's... and thats is what produces the 500 EOF's. interesting!
[reply]
[d/l]

Back to Seekers of Perl Wisdom