2ge has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I'm using LWP::Parallel and it works good for me. But I have one curious problem - I want change PUA agent for every request. But I don't know how to to that. Result of this code are 15 txt files, but, content is not what I expect. I'd like to have in TXT something like this:
"HTTP_HOST: 2ge.mine.nu" in every file
"HTTP_USER_AGENT: My agent x" where x is number 0-15
Just run this piece of code and you will see.
use strict; use warnings; use LWP::Parallel::UserAgent; my $http = 'http://2ge.mine.nu/headers.php'; my $max_hosts = 10; #registration of PARALLEL USER AGENT my $pua = LWP::Parallel::UserAgent->new(); $pua->in_order (0); # handle requests in order $pua->duplicates(0); # ignore duplicates $pua->timeout (10); # in seconds $pua->redirect (0); # follow redirects $pua->max_hosts ($max_hosts); # connect in parallel server $pua->max_req ($max_hosts); # max req per host my $counter = 0; my @agents = map "My agent $_", 1..15; for (my $i=0; $counter<@agents; $i++) { my @reqs_l = (); for (my $j=0; $j<$max_hosts; $j++) { push( @reqs_l,HTTP::Request->new('GET', $http) ); } my $reqs = \@reqs_l; #this could be better? foreach my $req (@$reqs) { if (defined $agents[$counter]) { $pua->agent("$agents[$counter]"); print "Registering agent: ".$counter."\n"; if ( my $res = $pua->register ($req) ) { print STDERR $res->error_as_HTML; } } $counter++; } my $entries = $pua->wait(); my $cnt = 0; foreach (keys %$entries) { print "WRITING content to $cnt.txt\n"; open(OUT,">$cnt.txt"); my $res = $entries->{$_}->response; print OUT $res->content( ); close OUT; $cnt++; } }

Replies are listed 'Best First'.
Re: LWP::Parallel in other way
by PodMaster (Abbot) on Nov 01, 2004 at 14:54 UTC
    But I have one curious problem - I want change PUA agent for every request. But I don't know how to to that.
    It may sound funny, but which part don't you know how to do? What do you think you'd have to do?

    PS - Please think about it for a minute, then answer my questions, this is going towards an actual solution.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

      Thanks for answer Chalamun :)

      I don't know which part of my code I should change, to work it like I expect, I don't know why it is not working now, so if you can, please help, and write correct snippet of code. Thanks.
        Oh come on, you're not even trying. Which part of the code is responsible downloading (HTTP request)?

        MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
        I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
        ** The third rule of perl club is a statement of fact: pod is sexy.