NodeReaper has asked for the wisdom of the Perl Monks concerning the following question:
See also Help with web crawling above.