in reply to Re^2: My first socket program is SLOW?
in thread My first socket program is SLOW?

I'm starting to think the best solution (since I'm actually trying to fetch about 6 pages and then snatch stuff out of them with regex's) would be to perform the requests simultaneously rather than one after another.

It's simple with threads:

#! perl -slw use strict; use threads; use threads::shared; use LWP::Simple; my @urls = qw[ http://news.bbc.co.uk/1/hi/default.stm http://q-lang.sourceforge.net/qdoc.html file://localhost/c:/perl/html/index.html http://search.cpan.org/~karasik/IO-Lambda-1.02/lib/IO/Lambda.pm http://www.ic.unicamp.br/~meidanis/courses/mc336/2006s2/funcional/L-99 +_Ninety-Nine_Lisp_Problems.html ]; my @threads; my %results :shared; for my $url ( @urls ) { push @threads, async{ printf "%d : fetching $url\n", threads->tid; $results{ $url } = get $url and printf "%d : got %d bytes\n", threads->tid, length $results{ $url } or warn "failed to get $url"; }; } print "Waiting for threads"; $_->join for @threads; print "threads completed"; ### process the content here. print "$_ : ", length $results{ $_ } for keys %results;

Output:

>t-get.pl 1 : fetching http://news.bbc.co.uk/1/hi/default.stm 2 : fetching http://q-lang.sourceforge.net/qdoc.html 3 : fetching file://localhost/c:/perl/html/index.html Waiting for threads 4 : fetching http://search.cpan.org/~karasik/IO-Lambda-1.02/lib/IO/Lam +bda.pm 5 : fetching http://www.ic.unicamp.br/~meidanis/courses/mc336/2006s2/f +uncional/L-99_Ninety-Nine_Lisp_Problems.html 3 : got 378 bytes 5 : got 56912 bytes 4 : got 65372 bytes 1 : got 76380 bytes 2 : got 1244518 bytes threads completed http://news.bbc.co.uk/1/hi/default.stm : 76380 http://www.ic.unicamp.br/~meidanis/courses/mc336/2006s2/funcional/L-99 +_Ninety-Nine_Lisp_Problems.html : 56912 http://search.cpan.org/~karasik/IO-Lambda-1.02/lib/IO/Lambda.pm : 6537 +2 file://localhost/c:/perl/html/index.html : 378 http://q-lang.sourceforge.net/qdoc.html : 1244518

Though if the list of urls to fetch grows much beyond a dozen or so, you'd want a slightly more sophistcated version that metered the number of concurrent gets and re-used the threads. That's also quite simple to write, but requires a little more thought.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."