A few possibilities. Here are two. The first is a minor change to the code:
my $root;
insert( \$root, 5 );
insert( \$root, 3 );
#...
sub insert {
my ($treeref, $value) = @_;
my $tree = $$treeref;
unless ($$treeref) {
$tree = {};
#...
$$treeref = $tree;
}
#...
This just makes it explicit that the variable being passed in (by reference) may be modified.
Another possibility is to have a "create tree" function for when the tree doesn't exist yet, and returns the initialised tree, possibly with the first value being required. The insert function would then need to tell if the tree is pristine and not yet populated (if the first value isn't required during create) to determine whether a new node was needed or not. The create-tree function would construct the tree - so we could call it a constructor. And we're basically at OO, all you have to do is bless the return :-)