I think that this autodetection is only going to be confusing to the user. Just provide two binaries - xxx-cli and xxx-gui, let the user decide what to run. Otherwise you're following the famous road paved with good intentions.

But to the actual point of the question - seriously, why not use a simple Tk test stub (although I would actually fork() off a child to run it).

Once again, though, be warned - people may hate this feature.