See the replies you got to your problem in Building a Spidering Application.

Maybe you want to reduce your problem by eliminating WWW::Mechanize from the picture?