in reply to LWP slow downloads on windows
This probably won't go down well with some, but mod:;//LWP seems to be badly broken on Win32. The following code downloads the same file from CPAN using 3 different methods; LWP::Simple::get(), LWP::UserAgent::get(), and Win32::Internet::FetchURL():
#! perl -slw use strict; use Time::HiRes qw[ time ]; use LWP; use LWP::Simple; use Win32::Internet; $|++; my $url = 'http://www.mirrorservice.org/sites/ftp.funet.fi/pub/languag +es/perl/CPAN/authors/id/J/JD/JDB/Win32-Internet-0.084.tar.gz'; my $start1 = time; my $file1 = get( $url ); printf "\nLWP::Simple took %7.3f seconds\n", my $time1 = time() - $sta +rt1; print "Size: ", length $file1; printf "Transfer rate: %5.2f bytes/sec\n", length( $file1 ) / $time1; my $inet = new Win32::Internet; my $start2 = time; my $file2 = $inet->FetchURL( $url ); printf "\nWin32::I took %7.3f seconds\n", my $time2 = time() - $start2 +; print "Size: ", length $file2; printf "Transfer rate: %5.2f bytes/sec\n", length( $file2 ) / $time2; my $ua = LWP::UserAgent->new; my $start3 = time; my $resp = $ua->get( $url ); printf "\nLWP::UA took %7.3f seconds\n", my $time3 = time() - $start3; + ## Corrected [Mr Mischief]++ my $file3 = $resp->content; print "Size: ", length $file3; printf "Transfer rate: %5.2f bytes/sec\n", length( $file3 ) / $time3;
The upshot is that LWP::Simple::get() was twice as fast and LWP::UserAgent::get(). But the native API wrapper is 25 times faster than LWP::Simple & LWP::UserAgent!
I could not believe the difference using the native API made--it HAD to be an error! Didn't it?-- but I've run this a dozen times now. I tried altering the ordering of the downloads to check that there was no caching involved; I re-booted and killed every process except those required to allow the system to run. And the following figures are pretty average for the results I've seen:
Uodated: figures for corrected benchmark. Mr Mischief++
C:\test>Win32Itest.pl LWP::Simple took 14.703 seconds Size: 64230 Transfer rate: 4368.46 bytes/sec Win32::I took 0.639 seconds Size: 64230 Transfer rate: 100515.50 bytes/sec LWP::UA took 14.796 seconds Size: 64230 Transfer rate: 4340.89 bytes/sec
Can anyone out there confirm these figures? Someone with a decent speed internet connection?
So, I tried wget:
C:\test>wget http://www.mirrorservice.org/sites/ftp.funet.fi/pub/langu +ages/perl/CPAN/authors/id/J/JD/JDB/Win32-Internet-0.084.tar.gz --02:17:39-- http://www.mirrorservice.org/sites/ftp.funet.fi/pub/lang +uages/perl/CPAN/authors/id/J/JD/JDB/Win32-Internet-0.084.tar.gz => `Win32-Internet-0.084.tar.gz' Resolving www.mirrorservice.org... 212.219.56.138, 212.219.56.139, 212 +.219.56.153, ... Connecting to www.mirrorservice.org|212.219.56.138|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 64,230 (63K) [application/x-gzip] 100%[==============...==============>] 64,230 4.51K/s ETA 0 +0:00 02:17:54 (4.60 KB/s) - `Win32-Internet-0.084.tar.gz' saved [64230/6423 +0]
Conclusion: if you use win32, and regularly download large files, seriously consider using the Win32::Internet module, because unless someone can explain my mistake, it really is remarkably quick.
|
|---|