Re: Site Crawler

This works great for yahoo.com or other generic sites. ... However, when I try it on my site, it fails to connect. My web host told me that they know of the problem, but they don't know why it doesn't work.

You don't mention whether this crawler is running on your desktop or form your ISP. If your script (or the networking layer beneath it) is having problems resolving the name of your host, where you're running makes a difference.

If you are running on your desktop, and can bring your site up in a browser, you should be able to bring it up via LWP.

If you are running the script on your ISP's box, and cannot access your site, there's probably a DNS problem at your ISP. If you have shell access there, try % ping www.whateveryoursitenameis.com If that fails, have a chat with your ISP again.

Comment on Re: Site Crawler Download Code

Replies are listed 'Best First'.
Re: Re: Site Crawler by Frisbeeman (Initiate) on Jun 13, 2002 at 20:47 UTC
Thanks for the reply. I'm running it from the isp, and do have shell access. I tried to ping our domain and it didn't work. I'll see what the isp thinks about that.	[reply]