Why TERM=xterm or TERM=linux breaking？

I don't know, I use TERM=dumb all the time.

Could be some setting of yours, bug in xterm/linux, bug in perl5db.pl, bug in whatever readline you're using (like this bug in Term::ReadLine) ... never had a real need for that stuff