Two ideas:

1) a csv is usually pretty small, not? To small to get accurate timing?
2) Do you use an active or passive session? Maybe the switching takes some time, while your ftp client uses passive right away.

I can't help you more, never used Net::FTP. Good luck...

   wouter