I don't have much experience with Mojo::UserAgent, but the Mojo Cookbook has examples for both non-blocking and blocking concurrency. For a simple test I got blocking concurrency with promises working. I built a small test server to avoid any webmaster complaints and easily read and did a little parsing on about 100 fetched URLs per second.
I put a 1 second delay into page delivery and performance seemed to depend on web server configuration. To get good performance I configured for 100 workers restricted to 1 connection each. So with a server named dos-serve-sp.pl I ran:
./dos-serve-sp.pl prefork -w 100 -c 1
Test server:
#!/usr/bin/env perl
use Modern::Perl;
use Mojolicious::Lite;
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
# cache page to echo
my $res = $ua->get('www.software-path.com')->result;
if ($res->is_error) { say $res->message }
elsif (not $res->is_success) {
die "Unknown response from Mojo::UserAgent"
}
$res->dom->at('head')->append_content(
'<base href="http://www.software-path.com">'
);
get '/' => sub {
my ($c) = @_;
sleep 1;
$c->render(text => $res->dom->to_string, format => 'html');
};
app->start;
Test client with blocking concurrency from promises:
#!/usr/bin/env perl
use Modern::Perl;
use Mojo::UserAgent;
my @all_url = ('http://127.0.0.1:3000/') x 200;
my $concurrent_load = 100;
my $ua = Mojo::UserAgent->new;
while (@all_url) {
my @concurrent_read = map {
$ua->get_p($_)->then(sub {
my $tx = shift;
my $result = $tx->result;
if ($result->is_success) {
say $result->dom->at('title')->text
}
else {
say $result->is_error ? $result->message :
"Unknown response from Mojo::UserAgent";
}
}) # end ->then sub
} splice @all_url, 0, $concurrent_load;
Mojo::Promise->all(@concurrent_read)->wait;
}
|