Use timeout to distinguish \e from Arrow, but use firm values after that. E.g. if Arrows are 3 chars long, read 1 char, if \e use action for \e which is read 1 char with timeout (timeout means real action for \e), if ok (and starts Arrowsequence) read last char without timeout.