in reply to Spidering websites

Try wget.

In Section Seekers of Perl Wisdom