Maybe there is some insight which can be taken from perl6-meditations/notations?

Signatures in Perl 6 are so expressive that I just use them unchanged in the documentation.

But I think using them in the documentation of a Perl 5 module is rather confusing. Simple things like optional parameters (indicated by a question mark after the parameter) might work, but there are things that have quiet a mismatch (for example you get checking and binding semantics similar to Perl 5 prototypes, but contrary to prototypes the parsing is not affected).

To return to your original question I'd suggest you take a look at the split documentation. It simply lists all possible call syntaxes, using upper case letters for metasyntactic elements.

I find that much easier to understand than [...] for optional arguments, which I too find confusing. They are a standard in unix man pages, but in Perl I don't like them, for the same reasons as you do.