Several suggestions:
- If you have a large number of secure file-transfers to do to remote locations, VPN is a great thing to use. A VPN-enabled router at home-office can connect to a VPN-enabled router in each remote site, using digital certificates, and when you do that, everything that you send down that pipe will automatically be very secure. Yet, this security is completely transparent to the clients. You might need a more-expensive router at home-office, but every router you get at any office-supply store these days is going to have VPN capability built-in.
- It's fine to use threads to parallelize file transfers, but you should never have “one thread per request.” Instead, visualize a thread as being just a worker-bee:
- Each worker-bee removes a work-request from a queue, carries it out, and then loops to get another request from the queue. When no more requests can be found (after a reasonable timeout), the thread politely dies.
- You should devise the system so that the number of threads can be easily set, and it will be based on the capacity of your I/O hardware (and the effective bandwidth of your network).
- With experimentation, you'll find a good balance of number-of-threads. Generally speaking, you'll see that performance improves up to a point, then it starts to degrade in a much-worse-than-linear fashion; the so-called “hitting the brick wall effect.”
- If a thread isn't able to complete a request, such as “nobody at the other end seems to be answering the phone,” the thread could respond to this problem by marking-up the request record in some way and throwing it back onto the queue. Then it grabs the next request from the same queue and keeps going. Requests that fail repeatedly would eventually have to be discarded.
As usual, there is a lot of support for this sort of thing out there already in CPAN. Be creative in your search.