Hmmm, I think you're talking about this:

$ ls -a
.  ..
$ touch a_file
$ ls -a
.  ..  a_file
$ echo a_*
a_file
$ echo b_*
b_*
[download]

This is how shells (or at least Bash) expand globs... if the glob can't be expanded it remains as is. There is a nullglob option in the shell that can fix that...

Is my only choice to stick that exact same directory listing into a var and look for that pattern and remove it ?

Yeah, 'remove' sounds good.