There are no easy answers for this, I wrote a http crawler once and it wasn't simple... What I'd suggest is switch LWP to HTTP::GHTTP, and see an existing module / script, for example HTTP::SimpleLinkChecker or so.