Using threads is only useful if you need to communicate between threads in real time. Otherwise fork-&-exec is a better solution. Why? Various drawbacks to threads, like possible memory gains over time, if a thread dies, it takes down the whole batch, and other inefficiencies. Reusing threads and the downloader packages will help in a long running threaded program. You might want to look at Reusable threads demo .

A forked solution is good, because all memory is released back to the system when the forked process ends. See LWP::Parallel::UserAgent and Parallel::ForkManager. You can also search for scripts using these on groups.google.com.

Anyways, before you think threads have solved your problem, watch it's memory usage as it runs. If it slowly gains, you have a problem with it.