Re^9: Parallel downloading under Win32?

Okay, I did three runs using the list of IDs you provided (63.6 MB):

-T=4: 6:20 - 171 KB/s.
-T=8: 3:54 - 276 KB/s.
(This is absolutely inline with my maximum throughput expectations for my connection.)
-T=16: 4:47 - 226 KB/s

By no means definitive, but sufficient to give me no reason to change my mind that 2 threads per core will usually give the best throughput. You might consider lowering the number of threads you run and see if it doesn't improve your throughput also.

One aside: If you have contact with the webmaster, you might suggest that he return a non-200 return code for unfound id's instead of returning 200 and a file containing: "Can't find that type". He explicitly asks people to not continually request non-existant data. That goal would be far easier to achieve if he did his bit by returning meaningful status codes.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Comment on Re^9: Parallel downloading under Win32? Select or Download Code

Replies are listed 'Best First'.
Re^10: Parallel downloading under Win32? by Xenofur (Monk) on May 02, 2009 at 17:51 UTC
Sorry for the delay, didn't find any time yesterday to do this. Either way, I can't reproduce your results. Here's the benchmark results on my end: $T=4 : 264 wallclock secs ( 3.80 usr + 2.09 sys = 5.89 CPU) -> 0.24 +MByte/sec $T=8 : 151 wallclock secs ( 5.31 usr + 1.89 sys = 7.20 CPU) -> 0.43 +MByte/sec $T=12 : 117 wallclock secs ( 6.55 usr + 2.41 sys = 8.95 CPU) -> 0.55 +MByte/sec $T=16 : 104 wallclock secs ( 7.73 usr + 2.83 sys = 10.56 CPU) -> 0.62 +MByte/sec $T=20 : 102 wallclock secs ( 8.36 usr + 2.80 sys = 11.16 CPU) -> 0.63 +MByte/sec $T=24 : 100 wallclock secs (10.11 usr + 3.00 sys = 13.11 CPU) -> 0.65 +MByte/sec $T=28 : 103 wallclock secs (11.89 usr + 3.02 sys = 14.91 CPU) -> 0.63 +MByte/sec $T=50 : 103 wallclock secs (18.89 usr + 4.30 sys = 23.19 CPU) -> 0.63 +MByte/sec [download] Right now I'm thinking that the difference between us is either the power of my laptop, your router or just plain your internet connection. Also, thanks for the suggestion, I'll forward it. :)	[reply] [d/l]

Replies are listed 'Best First'.

Re^10: Parallel downloading under Win32?
by Xenofur (Monk) on May 02, 2009 at 17:51 UTC

$T=4  : 264 wallclock secs ( 3.80 usr + 2.09 sys =  5.89 CPU) -> 0.24 
+MByte/sec
$T=8  : 151 wallclock secs ( 5.31 usr + 1.89 sys =  7.20 CPU) -> 0.43 
+MByte/sec
$T=12 : 117 wallclock secs ( 6.55 usr + 2.41 sys =  8.95 CPU) -> 0.55 
+MByte/sec
$T=16 : 104 wallclock secs ( 7.73 usr + 2.83 sys = 10.56 CPU) -> 0.62 
+MByte/sec
$T=20 : 102 wallclock secs ( 8.36 usr + 2.80 sys = 11.16 CPU) -> 0.63 
+MByte/sec
$T=24 : 100 wallclock secs (10.11 usr + 3.00 sys = 13.11 CPU) -> 0.65 
+MByte/sec
$T=28 : 103 wallclock secs (11.89 usr + 3.02 sys = 14.91 CPU) -> 0.63 
+MByte/sec
$T=50 : 103 wallclock secs (18.89 usr + 4.30 sys = 23.19 CPU) -> 0.63 
+MByte/sec
[download]

[reply]
[d/l]