Unfortunately I think you're right, these are just a few additional thoughts.
that's unfortunate, prototypes don't seem to influence precedence at all
Nitpick: They do influence precedence/parsing, just not in the way you want - from Prototypes:
sub mygrep (&@) mygrep { /foo/ } $a, $b, $c
sub myrand (;$) myrand 42
... treated specially by the parser. mygrep() is parsed as a true list operator, myrand() is parsed as a true unary operator with unary precedence the same as rand() ...
$ perl -MO=Deparse,-p -e 'sub myrand (;$) {} rand 123 / rand 456; myra
+nd 123 / myrand 456'
sub myrand (;$) {}
rand((123 / rand(456)));
myrand((123 / myrand(456)));
$ perl -MO=Deparse,-p -e 'sub mygrep (&@) {} grep {$A} + grep {$B} 1;
+mygrep {$A} + mygrep {$B} 1'
sub mygrep (&@) {}
grep( {$A;} grep( {$B;} 1));
mygrep(sub {$A;}, mygrep(sub {$B;}, 1));
... so builtins do basically behave the same way in terms of precedence - though as of 5.34 there isn't a builtin with a prototype of (&) or even (&@); grep has no prototype, which indicates it is treated specially by the parser (which is why I had to add the 1 after the above grep lest it be a syntax error instead of Not enough arguments).
Update: Relevant: Named Unary Operators