Why can't you get the port number before trying to connect? Untested:

use Net::FTP;  

$ftpd = Net::FTP->new('windows system') or die "Can't connect to 'dest
+': $!";  
$ftpd->login('anonymous') or die "Can't login to 'dest'";  

$ftpd->ascii() or die "Can't set ASCII mode: $!";  

$port = $ftpd->pasv or die "Can't put FTP host in passive mode: $!";  
print $port;

$ftpf = Net::FTP->new('unix system', port => $port) or die "Can't conn
+ect to 'from': $!";  
$ftpf->login('anonymous') or die "Can't login to 'from'";  
$ftpf->ascii() or die "Can't set ASCII mode: $!";  

$rfile = '/pub/swdistrib/conf/os.conf';  
$ftpf->retr($rfile) or $ftpf->ok or die "Can't retrieve '$rfile': $!";
+  
$sfile = '/swdistrib/os.conf';  
$ftpd->stou($sfile) or die "Can't store '$sfile': $!";  

$ftpd->pasv_wait($ftpf) or die "Transfer failed: $!";  

$ftpd->close() && $ftpf->close() or die "Can't close connections: $!";
+  
$ftpf->quit() && $ftpd->quit() or die "Can't quit ftp connections: $!"
+;
[download]

Update: removed redundant ascii call