As merlyn alluded to above, you need to add an RGB to luminance conversion. There is no one right way to do this, but the one used in American TV broadcasting is:

   Luma (gray value) = 0.299*Red + 0.587*Green + 0.114*Blue

Reference: Keith Jack, Video Demystified, 2nd Ed. (HighText Publications)