in reply to Re: Quicker way to batch grab images?
in thread Quicker way to batch grab images?
Another way, and similar to Re: use LWP::Simple slows script down., is to use Parallel::ForkManager & wget|curl (same caveats apply as in that sub thread).
A (useless) data point: With that combination and 6 sub-processes, I was able to download 100+ files around 1-10 MB each ridiculously in much less time -- did not track actual numbers for comparison -- than serial download. A single download takes 2 s to 5 s most of the time; some of worse cases are ~15 s.
|
|---|