I appologize, but your original post stated that you were looking for a link spider, and you did not state that for the problem you were trying to solve such behaviour would be overkill. You instead presented an alternative solution, which did not seem to fit the role of spider, as I understood it, because it did not parse the page and follow subsequent links. This may support the Reading the same text and getting a different impression thread.

Personally, I have done webserver support for many years (since 1995 ... my first server migration was so we would have support for software virtual servers and SSL), and keep a number of tools on hand for testing. For quick tests, I typically either just bring them up in a web browser:

$ netscape "url_goes_here"

or for times when the hosts aren't in DNS yet:

$ telnet server_ip_address
GET /url_path HTTP/1.0
Host: url_hostname

[download]

As another alternative, I have the respective content owners check their websites, while I keep an eye on the webserver's error log:

$ tail -f /path/to/webserver/error_log

You could also run wget against the site to begin the spidering, while you watch the logs, if you don't have linklint.

I have done a number of large scale server migrations, and the only time that we have ever had a problem with the data migration that wasn't caught in our migration testing, it was because we did not sample a large enough number of the pages. (one of the shell scripts to get all of the files that had been modified since the last tape backup generated too long of a list in one of the user's directories, which resulted in too long of a list sent to tar, which failed silently). And of course, the files that were missed were from a message board for a distance learning program ... so I spent the next two days consolidating the posts between the two servers.

Your original message also stated that the time savings were over finding a suitable program, and did not mention that you had a slow link, or were concerned with the time to get up to speed with the input parameters (although, a basic test with linklint is very simple). I admit that using Google for link spider pulls up crap, but this is one of those times where Yahoo does well. (okay, not on the original search, but it recommends 'linkspider', which has useful info.) Also, the search terms 'link checker' and 'link validator' both return useful results from Google.

I apologize if you took offense to my first reply, but I intended it to be constructive, and point you towards other tools that might be useful should you perform a similar migration again, and that your code sample didn't act as a link spider, which I interpreted from your message that you had intended it to serve in that role.

Update:: I suck at spelling.

In reply to Re^3: Reinventing the wheel by jhourcle
in thread Reinventing the wheel by bageler

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.