Since you are asking this on perlmonks, I assume you actually want to write the spider yourself, right? Check out O'Reilly's Perl & LWP. It discusses how to use the LWP bundle of modules to write web clients including spiders. It also discusses some of the pitfalls of spiders and how to avoid them. An invaluable resource if you hope to automatically download and parse HTML.

HTH