ajmcello has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I have a question regarding multiple http requests. I'd like to use Mojo, since I'm already taking advantage of Mojo::Pg, but it is not absolutely necessary. I have tried to do parallel http requests using perl threads, and came close, but have given up. Fork seems like it would take up too much of my resources.

Ideally, I want to go to 100 URLs at a time, and not wait for the previous one to finish before going onto the next URL. Then grab the next 100 and eventually go on to my function of the code.

I stole this from an earlier post, it is close, except that it creates a $tx transaction for each URL. Ideally, I want $tx to be something like $tx = "site1 site2 site3 site4" or something along those lines up to 100. 

use Mojo::Client;
use Mojo::Transaction;

my $client = Mojo::Client->new;

my $tx  = Mojo::Transaction->new_get('http://labs.kraih.com');
my $tx2 = Mojo::Transaction->new_get('http://mojolicious.org');
$tx2->req->headers->expect('100-continue');
$tx2->req->body('foo bar baz');

$client->process_all($tx, $tx2);

print $tx->res->code;
print $tx2->res->code;
print $tx2->res->content->file->slurp;

[download]

Thank you in advance

Replies are listed 'Best First'.
Re: Parallel HTTP requests using Mojo?
by kcott (Archbishop) on Dec 12, 2016 at 07:01 UTC

    You might like to take a look at Parallel::ForkManager. Its DESCRIPTION starts with:

    "This module is intended for use in operations that can be done in parallel where the number of processes to be forked off should be limited. Typical use is a downloader which will be retrieving hundreds/thousands of files." [my emphasis]

    It then continues with:

    "The code for a downloader would look something like this: ..."

    The code it shows uses LWP::Simple but could probably be adapted for your Mojo::* modules. I've never used Mojo so that's very much a supposition on my part (and not something I can help you with further).

    — Ken

[reply]
[d/l]
[select]
      I used Parallel::ForkManager and adapted to use Mojo.. Thanks much.
[reply]
Re: Parallel HTTP requests using Mojo? [OP moved]
by kcott (Archbishop) on Dec 12, 2016 at 06:40 UTC

    G'day ajmcello,

    You posted your "Parallel HTTP requests using Mojo?" question in "Perl Monks Discussion", which starts with:

    "This section is only for discussing issues pertaining to the PerlMonks web site. ..."

    I've moved it to "Seekers of Perl Wisdom", which starts with

    "If you have a question on how to do something in Perl, or you need a Perl solution to an actual real-life problem, or you're unsure why something you've tried just isn't working... then this section is the place to ask. ..."

    You don't need to do anything further regarding this beyond understanding why your post was moved. If you are in any doubt about where to post on this site, see "PerlMonks Sections" which describes the various sections and what's appropriate to post in them.

    Update: Added [OP moved] to the title to differentiate it from my second response which dealt with the actual question.

    — Ken

[reply]
Re: Parallel HTTP requests using Mojo? (Mojo::Client?)
by Anonymous Monk on Dec 12, 2016 at 08:39 UTC
    What is Mojo::Client?
[reply]
Re: Parallel HTTP requests using Mojo?
by LanX (Saint) on Dec 12, 2016 at 19:07 UTC
    > I stole this from an earlier post,

    which post?

    Cheers Rolf
    (addicted to the Perl Programming Language and ☆☆☆☆ :)
    Je suis Charlie!

[reply]

Back to Seekers of Perl Wisdom