lexthoonen has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I'm trying to do this:

$data = "http://www.peng.es/wetter/wetter.txt";

$lookup = HTTP::Request->new('GET' => "$data");
$response = $ua->request($lookup);

@lines = split (/\n/, $response->content);

and the result is a 404 from the peng.es site.

Although the file is accessible:
http://www.peng.es/wetter/wetter.txt

What could cause this?

Thanks.

  • Comment on how to troubleshoot a http::request 404 problem

Replies are listed 'Best First'.
Re: how to troubleshoot a http::request 404 problem
by Your Mother (Archbishop) on Feb 19, 2011 at 23:31 UTC

    Broken code could cause it. :) This works fine– 

    use strict;
use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $wetter = "http://www.peng.es/wetter/wetter.txt";
my $response = $ua->request( HTTP::Request->new('GET' => $wetter ) );
print $response->decoded_content, $/
__END__
<div align="center"><strong><span class="temp">15.3&deg;C</span></stro
+ng> <span class="resttemp">(59.5&deg;F)<br><font size="-2">23:25 - 19
+ Feb 2011</font></span></div>

    [download]

    When you have questions, you should post the exact code you’re running. What you posted doesn’t even create the UserAgent. Also, use strict and warnings, always.

    Update: WWW::Mechanize will make all this stuff more pleasant.

[reply]
[d/l]

      First of all, thanks for your answer, however, my result with your code is:

      <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /wetter/wetter.txt was not found on this server.<
+/p>
<hr>
<address>Apache/2.2.3 (CentOS) Server at www.peng.es Port 80</address>
</body></html>

      [download]

      Strange, no?

[reply]
[d/l]

        hmmm, that makes me think (at last...). I'll report back.

[reply]
Re: how to troubleshoot a http::request 404 problem
by roboticus (Chancellor) on Feb 19, 2011 at 22:48 UTC

    lexthoonen:

    I'm not a web developer, so this may be totally nuts, but the only thing that comes to mind for me is that the site may decide to ignore you because of the agent string. What happens if you set the user agent string to match Firefox or IE?

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

[reply]
      The user agent string clearly wasn't the problem here as YourMother shows. However, claiming to be different browsers is not "nutty".

      I've done maybe 1/2 dozen LWP programs and I've found that fiddling with the agent string by claiming to be something really stupid, like Netscape 4.76 can result in getting a returned page that is significantly easier to deal with than whatever the default is. I think that even in one case I was able to get a page without java script in it by just claiming to be "stupid". I'm sure the web developers had to code that as a special case, but evidently they did.

      Anyway some experimentation with claiming to be different browser types can sometimes produce interesting results. Of course this is completely site dependent. If you have the wrong level of browser for a site, usually you get a message saying that rather than a 404 (i.e. it will say something rather than remaining silent). Here's how to claim to be "stupid": 

      $ua->agent("Mozilla/4.76 [en] (Windows NT 5.0; U)");

      [download]
[reply]
[d/l]

        Marshall:

        Heh, heh! I didn't mean to imply that using a different agent string is a nutty idea. I've simply seen that suggestion in concert with this sort of request before. I gather that many web apps offer useless results if they don't see an agent string they recognize. (Such as "Oh, it's not Internet Explorer? Then instead of giving the browser the intended data, I'll send them to the Firefox page which doesn't do anything yet.")

        The nutty part is me offering a suggestion on code relying on modules I'm unfamiliar with. I intended the comment as a disclaimer.

        ...roboticus

        When your only tool is a hammer, all problems look like your thumb.

[reply]
Re: how to troubleshoot a http::request 404 problem
by Anonymous Monk on Feb 19, 2011 at 23:43 UTC
    and the result is a 404 from the peng.es site.

    So you knock on the door, and a woman opens the door and says 404.

    Maybe she is not Johnnys mom; maybe Johnny is at the dentist; maybe its Halloween and she is playing a trick; maybe you're dressed as a ghost, and she is scared; maybe Johnny is scared; ... maybe its all a dream

[reply]

Back to Seekers of Perl Wisdom