I wrote something like that using OpenGL for the display and ffmpeg as the decoder. I pipe the media files from ffmpeg to standard output as raw video frames and then push them onto the display via OpenGL. This allows fairly quick playback, but no audio control (due to the single output stream) and no backward seeking. On the upside, it was almost trivial to code and works.