You should not escape the $'s:

use warnings;
use strict;

my $Interface = 'bge1';
$Interface =~ s/[0-9]$/:$&/;
print "$Interface\n";

__END__

bge:1
[download]

Unrelated to your problem, but... Tip #1 from the Basic debugging checklist: use strict and warnings

Also, consider using $Interface =~ s/([0-9])$/:$1/; to avoid performance issues.