in reply to Re^2: Performance of FTP Transfer
in thread Performance of FTP Transfer
Your node got me wondering at what point forking more workers fails to improve performance. This is admittedly very environment-specific, but I found it interesting nonetheless. I put together the following simple script as a test:
Environment:
use strict; use warnings; use Net::FTP; my $kids; my @kids = (); my @fileList = glob("*.pdf"); open(OUT, ">ftp_test.out") or die; for $kids (1..30){ my @tempList = @fileList; my $start_time = time; while(my @files = splice(@tempList, 0, ($#fileList + 1) / $kids)){ my $pid = fork; push(@kids, $pid); if($pid == 0){ #kid &FTP_SUB(@files); exit; } } waitpid($_, 0) foreach (@kids); my $elapsed_time = time - $start_time; print OUT "Elapsed time with $kids kids:\t$elapsed_time seconds\n"; print "Elapsed time with $kids kids:\t$elapsed_time seconds\n"; } sub FTP_SUB { my @files = @_; my $start_time = time; my $end_time = undef; my $host = '...'; my $user = 'user'; my $pwd = 'pwd'; my $ftp = Net::FTP->new($host, Debug => 0); unless($ftp->login($user,$pass)){ warn "$$:\tCannot login " . $ftp->message; } unless($ftp->cwd("ftp_test")){ warn "$$:\tCannot change directory " . $ftp->message; } foreach my $file (@files){ unless($ftp->put($file)){ warn "$$:\tPut failed " . $ftp->message; } } $ftp->quit; $ftp = undef; }
__OUTPUT__ Elapsed time with 1 kids: 177 seconds Elapsed time with 2 kids: 92 seconds Elapsed time with 3 kids: 64 seconds Elapsed time with 4 kids: 56 seconds Elapsed time with 5 kids: 40 seconds Elapsed time with 6 kids: 34 seconds Elapsed time with 7 kids: 28 seconds Elapsed time with 8 kids: 27 seconds Elapsed time with 9 kids: 24 seconds Elapsed time with 10 kids: 23 seconds Elapsed time with 11 kids: 22 seconds Elapsed time with 12 kids: 20 seconds Elapsed time with 13 kids: 19 seconds Elapsed time with 14 kids: 18 seconds Elapsed time with 15 kids: 19 seconds Elapsed time with 16 kids: 17 seconds Elapsed time with 17 kids: 15 seconds Elapsed time with 18 kids: 17 seconds Elapsed time with 19 kids: 16 seconds Elapsed time with 20 kids: 16 seconds Elapsed time with 21 kids: 15 seconds Elapsed time with 22 kids: 14 seconds Elapsed time with 23 kids: 13 seconds Elapsed time with 24 kids: 12 seconds Elapsed time with 25 kids: 15 seconds Elapsed time with 26 kids: 13 seconds Elapsed time with 27 kids: 13 seconds Elapsed time with 28 kids: 13 seconds Elapsed time with 29 kids: 13 seconds Elapsed time with 30 kids: 13 seconds
You can see that the initial forks provided quite significant performance boosts. However, the improvement began to drop quickly once I was launching more than 7 children. Eventually there was no improvement whatsoever (though no cost, either).
Things to keep in mind:
|
|---|