in reply to Re^2: coderef for 1 .. 2?
in thread coderef for 1 .. 2?

"From the above code example, I believe this is an example of recursion. There is condition to exit (return unless my $url = shift @urls;) and $fetch is calling itself within the sub $fetch."

Yes, that's recursion. An exit stategy is a standard feature; all recursive functions should have this: without it, they will recurse infinitely (or, at least, to the extent system resources or configuration allow).

"... if the range operator has been removed ... Shouldn't the code process 1 request at a time until all the @urls have been processed?"

Yes, that's what I would have expected. I ran a few tests and all URLs are processed with any of these:

$fetch->() for 1 .. 2; # or $fetch->() for 1 .. 3; # or $fetch->() for 1 .. 4; # or $fetch->(); $fetch->();

But only one was processed with these:

$fetch->() for 1 .. 1; # or $fetch->();

I did some investigating. I will point out that I'm in no way a Mojo* expert; in fact, I wrote a very tiny application using Mojolicious::Lite about a month or so ago (and, until today, that was my only exposure to this family of modules).

I added lots of additional code to that cookbook example; mostly printing variable values or indications of where the code had got to. Eventually, I tracked this down to "delay" related code. After commenting out these two lines, a single "$fetch->()" processed all URLs:

### my $end = $delay->begin; ... ### $end->();

I was using Perl 5.26.0 and Mojolicious 7.46. Although the latter had only been installed about a month ago, I noticed that there had been five updates since then and 7.51 is the latest; the Changes file showed a number of delay-related comments, so I attempted to install the latest via cpan (I only got 7.50, but 7.51 is only a little over 24 hours old, so probably hasn't reached my CPAN mirror sites yet). I retested, with and without those two lines commented, and got the same results as before.

I checked the bug reports; none really seemed to address this issue; "Remove finish and error events from Mojo::IOLoop::Delay" looked like it was the only one that was related to delays.

I can't really spend any more time on this. The next place to look would probably be Mojo::IOLoop::delay(). There are others here with far more experience with the Mojo* modules; they may be able to provide much better answers than I have.

In the spoiler below, I've added my code with all the debug statements (and some additional code). You may find it interesting to track the process or for further investigations.

#!/usr/bin/env perl use strict; use warnings; use feature 'say'; use Mojo::UserAgent; use Mojo::IOLoop; say "PID: $$"; my @urls = ( 'mojolicious.org/perldoc/Mojo/DOM', 'mojolicious.org/perldoc/Mojo', 'mojolicious.org/perldoc/Mojo/File', 'mojolicious.org/perldoc/Mojo/U +RL', 'mojolicious.org/perldoc/Mojo/DOM', 'mojolicious.org/perldoc/Mojo', 'mojolicious.org/perldoc/Mojo/File', 'mojolicious.org/perldoc/Mojo/U +RL', ); # User agent with a custom name, following up to 5 redirects my $ua = Mojo::UserAgent->new(max_redirects => 5); $ua->transactor->name('MyParallelCrawler 1.0'); # Use a delay to keep the event loop running until we are done my $delay = Mojo::IOLoop->delay; my $fetch; $fetch = sub { my ($fetch_id) = @_; say "FETCH ID: $fetch_id"; say "PID: $$; URL count: ", 0+@urls, "; URLS: @urls"; # Stop if there are no more URLs #return unless my $url = shift @urls; return unless @urls; my $url = shift @urls; say "URL (pre delay): $url"; # Fetch the next title ### my $end = $delay->begin; say "URL (post delay): $url"; $ua->get($url => sub { my ($ua, $tx) = @_; say "\$ua->get(): \$ua[$ua] \$tx[$tx]"; say "$url: ", $tx->result->dom->at('title')->text; ### $end->(); # Next request $fetch->("INNER-$fetch_id"); }); say "AFTER \$ua->get()"; }; # Process two requests at a time $fetch->($_) for 1 .. 1; #$fetch->(1); #$fetch->(2); $delay->wait;

— Ken