Re: Proxy link rotation

Some time ago I writed LWP::UserAgent subclass, which main purpose is to automate proxy rotation for each request. It is not finished, not well tested and not documented yet, but you can try: LWP::UserAgent::Proxified
Simpliest variant of usage:

use LWP::UserAgent::Proxified;

my $ua = LWP::UserAgent::Proxified->new(
  proxylist => [
      ['http', 'https'] => 'http://10.0.0.1:1080',
      ['http', 'https'] => 'http://10.0.0.2:1080'
  ],
  proxyrand => 1, # choose random proxy for each request
# other lwp options goes here
);
[download]

Comment on Re: Proxy link rotation Download Code

Replies are listed 'Best First'.
Re^2: Proxy link rotation by kazak (Beadle) on Jan 27, 2012 at 21:14 UTC
I need to use 250-300 different proxies, all these proxies require authorization, also I need some special filter for disabling and enabling proxies in proxy-list (something like grey list),etc . Is there any mechanisms to implement these things within your class?	[reply]
Re^3: Proxy link rotation by OlegG (Monk) on Jan 28, 2012 at 04:43 UTC
I think you can `use strict; use LWP::UserAgent::Proxified; open( ROUTES, "<", "/etc/squid/repeater/lib/routes.cfg" ) or die "open: $!"; my @proxylist; while( <ROUTES> ) { s/#.//; next if /^(\s)$/; chomp; push @proxylist, http => "http://uname:passwd\@$_" if !/DISABLED/; } close(ROUTES); my $ua = LWP::UserAgent::Proxified->new( agent => undef, proxylist => \@proxylist, proxyrand => 1 ); # do the job` [download]	[reply] [d/l]