Mostly, I can discard based on URL - I've got a lot of links to pages with CGI variables set, which I know are duplicates or otherwise irrelevant.

So I'll have a look at doing something with LWP::UserAgent, and be careful and hope the recursion gremlins don't get me :)