I thought you were talking about your specific case, where I assumed you will have tests.

We package our installs with specific CPAN sources, so we never risk accidentally upgrading something. When we patch something, it's essentially the same as what you're doing -- an in-house change that works with one known version. The in-house code is marked with a different dist name than the CPAN dist. It would take a willful act to replace it with a different version.