The difference is negligible. Benchmark will frequently answer questions like these from a practical standpoint:

use Benchmark 'cmpthese';
use strict;
my $x = 0; our $y = \$x;
cmpthese( 10000000, {
    'braces'   => sub { return ${ $y } },
    'nobraces' => sub { return $$y },
});
[download]

My output is this:

Benchmark: timing 10000000 iterations of braces, nobraces...
    braces:  2 wallclock secs ( 1.87 usr +  0.00 sys =  1.87 CPU) @ 53
+47593.58/s (n=10000000)
  nobraces:  1 wallclock secs ( 1.87 usr +  0.00 sys =  1.87 CPU) @ 53
+47593.58/s (n=10000000)
              Rate nobraces   braces
nobraces 5347594/s       --       0%
braces   5347594/s       0%       --
[download]

If there is a difference, it's not terribly significant. For more complicated references, there may be some loss one way or another, but I doubt it would be much, if anything. Best as I could tell from using B::Deparse, the opcodes generated are different, but they likely work out to about the same thing (at least for hard refereneces... which you should be using exclusively). I find braces generally obnoxious, and so I avoid them when possible. However, they're a necessary evil, as demonstrated by the difference in y and z in the following:

my $x = [ [ 1, 2 ], [ 3, 4 ] ];
my @y = @$x->[ 0 ];
my @z = @{ $x->[ 0 ] };
[download]