I believe that, etymologically, the word is more related to ghost/phantom/illusion (although there is some overlapping semantics with spirit, of course) — in the sense that, under normal circumstances (if everyone abided by the traffic rules) Geisterfahrer should not exist.  So, in the unexpected and (hopefully) rare event that you happen to encounter one, you might at first actually feel like you can't trust your eyes, i.e. seeing a ghost or phantom...