I'm trying to make a really simple search engine for a site that looks up keywords in a dbm database. In order to populate the database, I am trying to write a crawler that will read in the meta-tag keywords and then crawl to all linked pages. I have a really simple test written that connects to a page and reads it's content. I found a useful tutorial here. Here's what I have so far: 
#!/usr/local/bin/perl

# Tests for site crawler / db creator

use LWP::Simple;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use HTML::LinkExtor;

$browser = LWP::UserAgent->new();
$browser->timeout(10);

$URL = 'http://www.yahoo.com/';

my $request = HTTP::Request->new(GET => $URL);
my $response = $browser->request($request);
if ($response->is_error()) {printf "%s\n", $response->status_line;}

$contents = $response->content();

print "Content-type: text/html\n\n";

print "<html>\n<head><title>Site Crawler</title></head>\n<body>";
print "<b>Here is the page's contents:</b><br>$contents";
print "</body>\n";
print "</html>";

[download]
This works great for yahoo.com or other generic sites. It shows yahoo.com with "Here is the page's contents:" at the top. However, when I try it on my site, it fails to connect. My web host told me that they know of the problem, but they don't know why it doesn't work. They told me to see if I could find a work around. I tried it in PHP as well, same deal. Anyway, while I struggle with the web host to fix this issue, I'm trying to find another way to do it. What are other ways to connect to a url and retrieve the data? Thanks for the help.

In reply to Site Crawler by Frisbeeman

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.