Part of your script is still running - that's why ssh doesn't terminate the connection:

...
my $pid =  fork();
...
[download]

You need to detach your child process from the parent process - maybe it's enough to close STDIN and STDOUT, but I'd use Proc::Daemon or what perldoc -q daemon says to start the program detached in the background:

use POSIX qw(setsid);

close STDIN; close STDOUT; close STDERR;
# or better, reopen them into some logfile

setsid();

fork && exit;

# We are now detached from ssh
[download]