Your "?" is modifying "()" instead of "*". You want

/ \[{ ( .*? ) }\] /x
[download]

/ \[{ ( [^\]]* ) }\] /x
[download]

Note that your strategy only works if all the content is two-level deep.