i like to read the ampersand in similar way to reading C, where it's literally "address of" operator.

It's never anything like the "address of" operator in Perl. In fact, it's often used to do the opposite (dereference a code ref).

so in that context the example line starting with ampersand would read: the address of an anonymous sub containing code/literal "color" is assigned the sub on the right hand side.

Which goes to show your way to read the ampersand is wrong. Nothing of the kind happens.