Unfortunately I think you're right, these are just a few additional thoughts.

that's unfortunate, prototypes don't seem to influence precedence at all

Nitpick: They do influence precedence/parsing, just not in the way you want - from Prototypes:

sub mygrep (&@)         mygrep { /foo/ } $a, $b, $c
sub myrand (;$)         myrand 42
[download]

... treated specially by the parser. mygrep() is parsed as a true list operator, myrand() is parsed as a true unary operator with unary precedence the same as rand() ...

$ perl -MO=Deparse,-p -e 'sub myrand (;$) {} rand 123 / rand 456; myra
+nd 123 / myrand 456'
sub myrand (;$) {}
  rand((123 /   rand(456)));
myrand((123 / myrand(456)));

$ perl -MO=Deparse,-p -e 'sub mygrep (&@) {} grep {$A} + grep {$B} 1; 
+mygrep {$A} + mygrep {$B} 1'
sub mygrep (&@) {}
  grep(    {$A;}    grep(    {$B;}  1));
mygrep(sub {$A;}, mygrep(sub {$B;}, 1));
[download]

... so builtins do basically behave the same way in terms of precedence - though as of 5.34 there isn't a builtin with a prototype of (&) or even (&@); grep has no prototype, which indicates it is treated specially by the parser (which is why I had to add the 1 after the above grep lest it be a syntax error instead of Not enough arguments).

Update: Relevant: Named Unary Operators