To prevent that, you need a negative look-ahead:

No, you don't.

$_ = "[[foo]] bar [[baz ] qux]] [quux]] fred [[waldo]]]";
say $1 while /\[\[([^]]*(?:\][^]]+)*)\]\]/g;
__END__
foo
baz ] qux
waldo
[download]

This uses some classical loop-unrolling, and is, IIRC, similar how Mastering Regular Expressions (first edition) matches C comments.