Usually one checks for the environment variable DISPLAY. If its defined and contains a nonempty string, you can assume a x-server is running. It is not completely reliable, AFAIK, but it is a good guess (for example, opening a window can fail even if you have DISPLAY set, on many systems you need additional authentication stuff).