Site seems to be blocking my LWP script

dchandler has asked for the wisdom of the Perl Monks concerning the following question:

Hello. I'm very new to perl and to LWP especially. I am trying to "getstore" information from a webpage but whenever I do this, the saved file is a 0k and completely blank. When I replace that same line with another website, I end up getting the source for that site. I have already enabled the pretend to be a browser thing $browser->agent("Mozilla/4.76 [en] (Win98; U)"). I think the webpage is thwarting me. What are some next steps to figure out if the webpage is blocking me or if my script is just messed up? The fact that it works for other sites makes me think the webpage is blocking me.

thanks **** THIS HAS BEEN RESOLVED **** thanks everyone!

Comment on Site seems to be blocking my LWP script Download Code

Replies are listed 'Best First'.
Re: question about LWP by Elijah (Hermit) on Aug 23, 2004 at 20:57 UTC
`#!/usr/bin/perl -w use strict; use LWP::Simple; my $url = "http://www.perlmonks.org"; my $file = "output.txt"; my $proc = getstore($url, $file); print $proc;` [download] This should tell you if you are getting the source or if there is an error connecting. It could be simply that the sight is down or the TTL is expiring before your script gets the source. Just change the value of $url to whatever sight is giving you issues. The return value of $proc will be the web server code for the connection attempt. A return value of 200 means successfull and a return value of like 500 or 403, or 404 are error codes and means sometyhing was wrong with the request and the web server could not or will not handle it. www.perlskripts.com	[reply] [d/l]
Re^2: question about LWP by dchandler (Sexton) on Aug 23, 2004 at 21:12 UTC
Thanks so much for helping me out. Unfortunately, when i use the exact lines you have I get a code of 200, but I still get a blank document. I have tested this on a reliable site (google) and can get their source. but I cannot get the page I've been trying to get.	[reply]
Re: question about LWP by xorl (Deacon) on Aug 23, 2004 at 20:55 UTC
First view the site in a normal web browser. If it displays fine then, find out exactly what data is being sent to the website by the browser and what exactly the browser is getting back from the server. It could be some silly JavaScript thing or it could be something like they're blocking your IP. Others around here can probably help more.	[reply]
Re: Site seems to be blocking my LWP script by Nkuvu (Priest) on Aug 24, 2004 at 00:31 UTC
** THIS HAS BEEN RESOLVED ** thanks everyone! How was it resolved? Your solution may help others with the same or similar problems...	[reply]