Maybe it's not the most useful concept (in my mindset) to bring across from C to perl with me...

Definitely, cause Perl's "&" never does anything similar to C's "&". Perl's "\" would be the closest to C's "&".

anyhow you can dereference a code reference in perl without the ampersand

$ref->(), although I don't see how the question relevant.

so any argument about what ampersand actually does in perl is debatable

Not really. It can be used to bypass prototypes (&func(...)), to call a func with parent's args (&func;), to dereference a code ref (&$ref) and surely more. None of them return an address (like C's "&") or anything similar (like Perl's "\").