Hi, I am having a little problem with LWP. This script checks links in a mysql database. Everything works ok, but as I was testing I noticed an issue. When it checks a url that no longer exists, but the server it formerly resided on has custom 404 error pages, the link checks as being valid. I guess what I want to do is only consider the url valid if it returns a 200. How can I modify the script to work like this? 
#!/usr/bin/perl 

use LWP::UserAgent; 
use DBI; 

$db_database = "db"; 
$db_uid = "user"; 
$db_pwd = "pass"; 
($ua = LWP::UserAgent->new)->timeout(20); #actually set timeout 


$dbh = DBI->connect ("DBI:mysql:$db_database".$mysqlsock, $db_uid, $db
+_pwd) or die("could not connect to db\n"); 
$sth = $dbh->prepare("SELECT url FROM files"); 
$sth -> execute(); 
$numrows = $sth->rows; 

$i = 0; 
$works = 0; 
$notworks = 0; 
print "\n\n"; 
#while (my $url = $sth->fetchrow_array) { 
while (defined(my $url = $sth->fetchrow_array)) { 
if(($ua->request(HTTP::Request->new('HEAD', $url)))->is_success()) { 
$validity = "link works"; 
$valid_update = $dbh->do("UPDATE files SET valid = 1 WHERE url = '$url
+'"); 
++$works 
} 
else { 
$validity = "link sucks"; 
$valid_update = $dbh->do("UPDATE files SET valid = valid + 1 WHERE url
+ = '$url'"); 
++$notworks; 
} 
++$i; 
print "$i of $numrows\n$validity\n$url\n\n"; 
} 
$sth->finish;

[download]

In reply to help with link checking by rmckillen

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.