You have to ask yourself what is slower...passing the array/hash, or the extra deference everytime you assess it.
Typically, passing by reference is a win, but now always.
Remember:
$foo{'bob'};
is faster than:
$bar->{'bob'}; # Slower, extra dereference
So if you have a small array/hash, and dereference it alot, code it both ways, and see what Benchmark tells you. I'm betting the difference will be small enough, you should be optimizing for readability instead.
Also, remember using reference allows you to modify the
actual variable, where normally you will just modify a copy. This, and not performance, will usually determine which you should use.