Verified on ActiveState v5.8.9 built for MSWin32-x86-multi-thread with

perl -MTerm::ReadKey -e "ReadMode 3; print ord ReadKey( 0 ) while 1"

Also obtained expected (non-swallowing) behavior on v5.10.1 (*) built for x86_64-linux-gnu-thread-multi.

No explanations come to mind. A quick read of Suffering from Buffering yielded no insight.