in reply to Recursive HTTP Downloads - without using WGET

If you are rejecting the URL based on the URL itself, then LWP::UserAgent may help you create something like what you describe. On the other hand, if you are rejecting the URL based on the contents of it, then the WGET method as you describe is how it would need to be done anyway.

--MidLifeXis

  • Comment on Re: Recursive HTTP Downloads - without using WGET

Replies are listed 'Best First'.
Re^2: Recursive HTTP Downloads - without using WGET
by Preceptor (Deacon) on Jul 16, 2012 at 07:36 UTC
    Mostly, I can discard based on URL - I've got a lot of links to pages with CGI variables set, which I know are duplicates or otherwise irrelevant.

    So I'll have a look at doing something with LWP::UserAgent, and be careful and hope the recursion gremlins don't get me :)

[reply]

In Section Seekers of Perl Wisdom