I knew you wouldn't. Because you obviously don't know how.

Demonstrate for your fellow Monks,

I already did 500+ times.

Meanwhile a search for your threaded code contributions turns up one hit. And there's no code in that either.

how throwing 500 threads at this problem is justifiable and necessary.

Simple math. You have 2000 SaaS servers from which to retrieve data, that takes them 10 seconds to prepare and return.

Using 4 threads: 2000*10/4 = 5000 seconds = 1hr 23 minutes elapsed.

Using 100 threads: 2000*10/100 = 200 seconds = 3.3 minutes elapsed.

Using 500 threads: 2000*10/500 = 40 seconds elapsed.

You need another demo?

[ 0:35:30.57] c:\test>t-head-urls -THREADS=4 urls.list
Processed 2040 urls
Took 424.354000 with 4 threads

c:\test>t-head-urls -THREADS=8 urls.list
Processed 2040 urls
Took 215.305000 with 8 threads

c:\test>t-head-urls -THREADS=16 urls.list
Processed 2040 urls
Took 123.232000 with 16 threads

c:\test>t-head-urls -THREADS=32 urls.list
Processed 2040 urls
Took 90.033000 with 32 threads

c:\test>t-head-urls -THREADS=64 urls.list
Processed 2040 urls
Took 88.804000 with 64 threads
[download]

You see where that's headed?

So now it is (way past) your turn. Ante up if you can.

Bumcombe!!

If I have, as you yourself have said, “2,000 SAAS servers from which to retrieve data, that takes each of them 10 seconds to prepare and return,” then I most-emphatically don’t require “an equal number of my processor threads to be sitting around, each of them waiting for one of those SAAS servers to answer me.“

If I am waiting to hear back from “2,000 SAAS servers,” then the one thing that I can absolutely count on is that:   it will not be the case that all 2,000 of them will reply to me within precisely the same millisecond.   There is, after all, one TCP/IP interface card on this box, therefore all of the datagrams will arrive one at a time.   Many, many milliseconds apart.   Therefore, it is entirely to be expected that just one process, on my end, will be able to handle the 2,000 separate responses that will arrive.

Y’know, BrowserUK, I am afraid that these days you spend so much time opposing what I have to say, on the assumption that during the course of 30 years in this business that I have learned absolutely nothing in the way of pragmatic and practical and worldly experience, that you seem to spend almost no time listening to whatever it is that I have to say.   And maybe, (durst I say it?) you really ought to “meditate” on that, for a little-biddy while.   It might just do you some good.   Because, quite frankly, I do not have the time, nor do I have the slightest inclination, to quarrel with you further on these points.

Good day, sir.

Therefore, it is entirely to be expected that just one process, on my end, will be able to handle the 2,000 separate responses that will arrive.

Oh yeh! Bollocks! Here is your favoured "select loop" (not implemented by me, but rather the best CPAN has to offer to ensure fairness), processing those same 2040 urls that 64 threads dealt with in 88 seconds:

#! perl -slw
use strict;
use Time::HiRes qw[ time ];
use LWP::Parallel::Useragent;
use HTTP::Request;

my $start = time;

my $pua = LWP::Parallel::UserAgent->new();
$pua->timeout( 10 );

$pua->register(  HTTP::Request->new( 'HEAD', "http://$_" ) ) while <>;

my $entries = $pua->wait;

printf "Took %.6f seconds\n", time - $start;


__END__
c:\test>pua-head-urls urls.list
Took 1333.616000 seconds
[download]

Yup! One thousand, three hundred and thirty three point six one six seconds.

For all your claims of 30 years of experience, and fine protestations, you know nothing. You just regurgitate what you think you've read as authoritative. And usually get it completely wrong.

Can you do better than the author of LWP::Parallel::UserAgent? I'm betting you will not even try. Just like you never have.

BTW: I'm not arguing with you. I'm simply telling you that you're wrong. The only counter-argument is proof. Ie. Runnable code that supports your erroneous statements. Ante up or shut up.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?