in reply to Re: Speeding things up -- LWP::Parallel
in thread Speeding things up
I don't have much experience with Mojo::UserAgent, but the Mojo Cookbook has examples for both non-blocking and blocking concurrency. For a simple test I got blocking concurrency with promises working. I built a small test server to avoid any webmaster complaints and easily read and did a little parsing on about 100 fetched URLs per second.
I put a 1 second delay into page delivery and performance seemed to depend on web server configuration. To get good performance I configured for 100 workers restricted to 1 connection each. So with a server named dos-serve-sp.pl I ran:
./dos-serve-sp.pl prefork -w 100 -c 1
Test server:
#!/usr/bin/env perl use Modern::Perl; use Mojolicious::Lite; use Mojo::UserAgent; my $ua = Mojo::UserAgent->new; # cache page to echo my $res = $ua->get('www.software-path.com')->result; if ($res->is_error) { say $res->message } elsif (not $res->is_success) { die "Unknown response from Mojo::UserAgent" } $res->dom->at('head')->append_content( '<base href="http://www.software-path.com">' ); get '/' => sub { my ($c) = @_; sleep 1; $c->render(text => $res->dom->to_string, format => 'html'); }; app->start;
Test client with blocking concurrency from promises:
#!/usr/bin/env perl use Modern::Perl; use Mojo::UserAgent; my @all_url = ('http://127.0.0.1:3000/') x 200; my $concurrent_load = 100; my $ua = Mojo::UserAgent->new; while (@all_url) { my @concurrent_read = map { $ua->get_p($_)->then(sub { my $tx = shift; my $result = $tx->result; if ($result->is_success) { say $result->dom->at('title')->text } else { say $result->is_error ? $result->message : "Unknown response from Mojo::UserAgent"; } }) # end ->then sub } splice @all_url, 0, $concurrent_load; Mojo::Promise->all(@concurrent_read)->wait; }
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^3: Speeding things up -- LWP::Parallel
by AlwaysSurprised (Novice) on Jun 16, 2018 at 23:54 UTC | |
by hippo (Archbishop) on Jun 17, 2018 at 09:35 UTC |