in reply to Using LWP::Parallel

Maybe i'm missing something, but in neither of the 2 replies posted so far do i see anything that seems to address wilstephens's specific question about limiting to amount of parrallellisssmmmm to a pecific number (his example was 10).

I've never used LWP::Parallel, but after a quick glance at the docs (the concept intrigues me) I notice this method for LWP::Parallel::RobotUA ... 

  $pua->max_req  ( 2);  # max parallel requests per server

[download]
but that's not quite what we're looking for.

I can't for the life of me see anything obvious here. I would guess you could do this by only registering 10 URLs at a time, getting them in parrallel, then registering another 10, etc. But this is wastefull of cycles. Some URLs will finish faster then others, but you're wiating for all 10 before proceeding to #11.
(I would guess there is a better way)

UPDATE: LWP::Parallel::UserAgent also seems to have a max_req method, as well as a max_hosts method -- so if you know that all of your URLs wer for different hosts, OR were all for the same host, you could use one of them to get your result, but i still don't see anyway to optimally fetch no more then 10 URLs a time without any pre previous knowledge of your URLs. method

Replies are listed 'Best First'.
Re: Re: Using LWP::Parallel
by mugwumpjism (Hermit) on Mar 11, 2002 at 19:44 UTC
    Actually, it's quite trivial to get the POE solution to fetch 10 documents at a time; you just create 10 sessions which continually (get a URL off a master list, then fetch that document).
[reply]
      I wouldn't call it trivial if people have no experience with POE. I for one didn't understand from your example how you were suggesting it would help him towards his goal of retrieving no more then 10 at a time.

      After reading the POE, POE::Session, & POE::Component::Client::UserAgent I *think* what you're suggesting is that the poster modifying 02multi.t so that this... 

      POE::Session -> create (
    inline_states => {
          _start => \&_start,
          _stop => \&_stop,
          response => \&response,
          _signal => \&_signal
      },
  );

      [download]
      becomes this... 
      for (my $i = 0; $i < 10; $i++) {
    POE::Session -> create (
        inline_states => {
              _start => \&_start,
              _stop => \&_stop,
              response => \&response,
              _signal => \&_signal
        },
     );
}

      [download]
      and change _start to only loop over a 1/10 of @urls.

      Does that sound about right?

      Frankly, i'm still not clear on why this can't be done in a more straight forward manner with LWP::Parallel directly, It has the functionality to limit the number of parrallel requests to an individual server -- OR to limit the number of different servers it sends requests to at the same time, ... why isn't there a more general way to limit the TOTAL number of parrallel requests?

[reply]
[d/l]
[select]
        Frankly, i'm still not clear on why this can't be done in a more straight forward manner with LWP::Parallel directly, ...

        Well, for a start you can't do anything while the fetch is happening.

        I've written a little script that keeps 10 threads; it's in my original post. As you can see, there's not much to it.

        (Event-based programming)++

[reply]

In Section Seekers of Perl Wisdom