Unfortunately, that didn't do it for me. I would guess that the <> operator has a higher precedence than the forced interpolation of the double quotes. On the plus side, specifying glob as rmgiroux points out works very well.
(secret: put the print statement before the unlink and it will display how many files it unlinked.)