in reply to Re: Split large file and upload to ftp server with multiple threads - SANITY CHECK
in thread Split large file and upload to ftp server with multiple threads

Independent circuits combined providing greater bandwitdh than a single circuit is correct. I wasn't sure how to reconstitue the files either, or if it was even possible with FTP.
  • Comment on Re^2: Split large file and upload to ftp server with multiple threads - SANITY CHECK

Replies are listed 'Best First'.
Re^3: Split large file and upload to ftp server with multiple threads - SANITY CHECK
by salva (Canon) on Oct 02, 2008 at 08:43 UTC
    Independent circuits combined providing greater bandwidth than a single circuit is correct
    Unless you have two different physical connections between the local and remote network (and that means two modems, two DSLs, or two whatever at the local side), you don't have two independent circuits.

    On the old days, when networks had lots of errors and packets dropped, using several TCP connections (all going over the same physical layer) to send data in parallel was an effective mean to increase throughput. But nowadays, that the networks are quite reliable, it doesn't make sense anymore, you will only get a marginal improvement (if any) on the transfer speed.

    Effective ways to reduce the transfer time are:

    • Use compression, if you are already using compression, use a better compressor (bzip2 is better than gzip and z7 is usually better than bzip2).
    • Sometimes your data can be represented in a more compact format. For instance, CSV is better than XML in this regard.
    • Can you use rsync?