I don't have any concrete ideas, but my co is using Net::FTP in production environments to do this sort of thing, and has never run across this problem. Currently running on Linux (SuSE), HP-UX, and AIX (clients), with HP-UX, AIX and Windows (IIS) as servers. My guess would be its an OS problem, where something in the setup isnt letting the closed ports time out properly.
Maybe you can show your client/server details, along with perl and Net::FTP versions?