First of all, this is about pseudo-hashes. This (now long gone) feature lets you access an array reference as if it was a hash reference.

So what does $aref->{key} actually do? Well, this: $aref->[$aref->[0]{key}]. That is, it fetches the first element of the array (which must be a hashref) and uses it to map the key to an array index, which is then finally looked up in @$aref.

The other piece of the puzzle is autovivification (also explained in perldoc perlref). This is what automatically creates references for you when you try to dereference something that doesn't exist. Example:

my $foo; $foo->[1] = 42;  # automatically sets $foo = [] before the
# assignment so you end up with $foo = [undef, 42] instead of
# dying with "Can't use undef as an ARRAY reference"
[download]

OK, now let's look at your code again. $a->{test} looks up "test" in { test => { val => 1 }} (the first element of @$a) and returns { val => 1 }. It then uses this reference as an array index, which converts it to a (very large) integer (the internal memory address of the hash).

$a->[0xDEADBEEF] obviously doesn't exist; the array only has two elements. But since you're dereferencing it (the ->{val} part), perl tries to autovivify a hashref for you. Unfortunately this involves resizing the array to have 0xDEADBEEF + 1 elements (the intervening elements are set to undef). At this point perl is running out of memory, as expected.