You need a trailing star in your glob pattern (it currently appears to have a trailing dash). Also, excluding all files with dashes is a bit of overkill. You can eliminate space-dashes by eliminating spaces:

for my $FN ( glob "$dir/".'*[\ \\?<>,`!@#$%^&*();+={}[]*' ) {
[download]

I can eliminate opening brackets, but when I try to escape a closing bracket, suddenly nothing matches.

Note that you could get the same effect with

for my $FN ( grep /[ \\?<>,`!@#$%^&*();+={}[\]]/, <*>)
[download]

Season to taste.