Part of what I was hoping to do do was test what throughput I could get via my limited bandwidth using mutiple threads and overlapped IO.

My intention was to download high proportion of PM and then produce a fully inverted index to it locally.

With the concern that there was never a time here when I could run the process without having an undue impact, I got permission to do my throughput testing at the weekend at another site where their peak usage is weekdays.