The presence of a dollar-sign is not the issue. The value of the variables at this point in the program is what's suspect:

my $a;        # undef
my $b = 1;

my @array = (
        [4, 3],
        [2, 1],
);

print $array[$b]->[$b]; # 1
print $array[$b]->[$a]; # Use of uninitialized value in array element,
+ but prints 2
print $array[$a]->[$b]; # Use of uninitialized value in array element,
+ but prints 3
[download]

The undefined value has a numeric value of 0, so it all "works" because of Perl's DWIM nature, but through the grace of warnings we know that something is amiss if we expect that our index variables should always have a numeric value.