the paths you do no want to expand are any where the scalar sigil $ is not immediately after the filepath separator. In which case you can allow these by matching for that construct.

my $fps = 'filepathseparator';
m!
$fps             # your file path separator
\Q$\E            # quoted scalar sigil (shell variable expansion)
\{?              # start shelling out for a return value ?
([^\W]+)         # your environment variable, not containing,
                 # one or more non-word characters
\}?              # end shelling out for a return value ?
!x
[download]

This should match both cases where the curly braces are or are not used while constraining the variable to starting immediately after the file_path_separator (no mid/end word sigils match), and not allowing the path word to contain any non word characters.