Strange. I tried it using the OO interface with named params, and it worked on three different ports for me. Try:

my $scp = Net::SCP->new( {"host" => 'server1.ca:3479', "user" => 'user
+1'} );
$scp->scp($source, $destination);
[download]

updated: added $scp->scp($source, $destination);