If you're pondering writing your own 'spider' using LWP, a handy tool you should familiarize yourself with is HTML::LinkExtor, which will extract all the links on a page for you.

But boy, oh boy, if you do this job right for sites of moderate complexity, you are going to learn a lot about complex data structures in perl =) Good Luck!