dogbert has asked for the wisdom of the Perl Monks concerning the following question:

hi fellow monks, I'm just getting into using LWP and I have a quick question. Is there a way to use more then one proxy or maybe even rotating proxies? and if so can someone point me to that information, as I have already read the man page for LWP and the LWP::UserAgent and all I have found is how to use one proxy. thanks

Height : varies depending on my speed relative to the observer Weight : is a sensation caused by the gravitational wrapping of time-space Age : time is only a persistent illusion

Replies are listed 'Best First'.
Re: LWP and proxies
by dws (Chancellor) on Apr 11, 2003 at 01:45 UTC
    Is there a way to use more then one proxy or maybe even rotating proxies?

    By "more than one" do you mean a chain of proxies? That'd be rather difficult. Even browsers don't support chains (except when only the first proxy in the chain is visible, and the rest are automagic).

    Rotating through a set of proxies should be too difficult. All you need to do is remember which you used last (or, alternatively, which one to use next), then invoke 

      $ua->proxy('http', $proxy);

    [download]
    at the appropriate time (where $ua is an LWP::UserAgent, and $proxy is the URL of the proxy to use).

[reply]
[d/l]
      thanks for the info. Now that you bring it up I guess I was looking for a way to rotate the proxies rather then use more then one at a time. I really need to watch the way I word things around here : )

      Height : varies depending on my speed relative to the observer Weight : is a sensation caused by the gravitational wrapping of time-space Age : time is only a persistent illusion

[reply]
Re: LWP and proxies
by jasonk (Parson) on Apr 11, 2003 at 02:40 UTC

    You could pretty easily rotate through proxies just by changing your proxy settings before fetching a page:

    my @proxies = (
    'http://proxy.com:1234',
    'http://someotherproxy.com:8080',
    'http://somerandomproxy.com:4123',
);
foreach(@proxies) {
    $ua->proxy('http', $_);
    $ua->get("http://myserver.com/page_of_ad_banners.html");
}

    [download]
    We're not surrounded, we're in a target-rich environment!
[reply]
[d/l]

Back to Seekers of Perl Wisdom