See my Babysitting an interactive CPAN update, although you may be able to just edit the .pm file that contains the settings instead. Perhaps CPAN::HandleConfig might be a good starting place for the latter.