You have to set 'vi' mode in your ~/.inputrc file. Such has worked for me in the Perl debugger, though I don't recall the details. The web suggests you try:

set editing-mode vi
set keymap vi-command
[download]