Your description and examples aren't enough to to deduce a regex from them. You should try to define precisely what you want to allow or not. This could look like that:

A valid ref spec is a string that starts 
with the literal 'element ', and is followed by

* Either a star '*' OR by m/-dir(ectory)? \S+/
                    OR by m/-file \S+/
* then by a label

A label is ...
[download]

Once you know what you actually want to match, you can translate that into a regex quite easily:

my $re = qr{
   element \s
      (
        \*
      | -dir(ectory)? \S+
      | -file \S+
      )
   # Label:
   \w+
}x;
[download]

You should also handle special cases special, ie don't try to munge them into your regex, but rather make an extra path to check for special cases.