Read Web Techniques Column 7 (October 1996) from merlyn. It presents a recursive link checker.

Christian Lemburg
Brainbench MVP for Perl
http://www.brainbench.com