I am not sure what exactly you are trying to do, but, with the start and end of line anchors, your regex will match only lines which have only exactly one character of the character class and nothing else. You may want to have at least have something like:

$string =~ s/^[ \t]+$//mg;
[download]

Or maybe you don't want the ^ and $ anchors. Or maybe you don't want the multi-line m modifier. Please explain more precisely what you want to achieve, and give an example input where it does not work the way you wish, together with the output that you are looking for.