I just ran into a similar problem with perlbrew and Term::ReadLine:Perl this week on an Ubuntu 19.04 box.

I had Term::ReadLine::Perl installed via cpan. This would break with a similar error as yours when I went into the debugger for any version installed with perlbrew.

After numerous attempts to install Term::ReadLine::Perl (or ::Gnu) in the perlbrew environments, I was finally able to resolve the issue. I needed to remove Term::ReadLine::Perl from cpan and then install my system's library. In my case, I installed libterm-readline-perl-perl