but I'm not sure why that doesn't work.

Because ReadKey "converts" inbound keycodes to VT100/termcap-compatible control sequences for 'portability' sake.

The full decision tree required to decode these sequences--at least those emulated on a win32 system--is:

#! perl -slw
use strict;
use Term::ReadKey;

ReadMode( 4 );

while( 1 ) {
    my $c = ReadKey( 0 );
    if( $c eq "\e" ) { # got an escape
        if( $c = ReadKey(-1) ) {
            ## If its an extended key they'll be another character in 
+the buffer
            if( $c eq '[' ) {
                # Could be an extended key, the next character to dete
+rmine which
                $c = ReadKey( -1 );
                if( $c eq "A" ) {
                    print "Got uparrow";
                }
                elsif( $c eq "B" ) {
                    print "Got downarrow";
                }
                elsif( $c eq "C" ) {
                    print "Got rightarrow";
                }
                elsif( $c eq "D" ) {
                    print "Got leftarrow";
                }
                elsif( $c eq "2" ) {
                    print "Got Insert";
                    $c = ReadKey( 0 ); ## Discard (useless) '~'
                }
                elsif( $c eq "3" ) {
                    print "Got Delete";
                    $c = ReadKey( 0 ); ## Discard (useless) '~'
                }
                elsif( $c eq "1" ) {
                    print "Got Home";
                    $c = ReadKey( 0 ); ## Discard (useless) '~'
                }
                elsif( $c eq "4" ) {
                    print "Got End";
                    $c = ReadKey( 0 ); ## Discard (useless) '~'
                }
                else {
                    print "Ignoring unknown extended key: '$c;";
                    next;
                }
            }
            elsif( $c eq "O" ) {
                ## Another set of extnded keys
                $c = ReadKey( -1 );
                if( $c eq 'y' ) {
                    print "Got PageUp";
                }
                elsif( $c eq 's' ) {
                    print "Got PageDown";
                }
                else {
                    print "Ignoring unknown extended key: '$c;";
                    next;
                }
            }
            else {
                print "Ignoring unknown extended sub-selector '$c'";
                next;
            }
        }
        else {
            # Just a simple escape, so quit
            print "Got Escape; exiting";
            last;
        }
    }
    elsif( $c eq chr(0) ) {
         print "Got a null";
    }
    else {
        print "Got '$c' ", ord( $c );
    }
}

ReadMode( 0 );
[download]

But note: that doesn't allow any access to the function keys; or ctrl/alt/shift combinations with the arrow and other navigation keys. It's really a rather useless module in that respect.

If you are using Windows, then the more useful module for keyboard (and mouse) handling is Win32::Console::Input(), which allows you access to all of the Windows defined virtual keycodes + the actual scancodes; but it is still quite a messy affair and totally non-portable,