The irc_public => \&on_public reference continues to point to the original subroutine even after the package that contained it is reloaded. (This is what lets you wrap subroutines by saying my $old = \&subname; *subname = sub { &$old }.)

As revdiablo suggests, pass a code reference that explicitly calls the subroutine by name so that the new definition will be found: irc_public => sub { &MyPackage::subname }.