I think you can't do it because of stubs

sub foo;  # stub

print("\&foo is ", defined &foo ? 'defined' : 'not defined', "\n");
print(__PACKAGE__->can('foo') ? 'Can' : 'Can\'t', " foo\n");

foo();

__END__
&foo is not defined
Can foo
Undefined subroutine &main::foo called.
[download]

Undefining the CODE slot removes the sub there, but not the stub. Apparently, defined checks for the sub, while can checks for the stub.