You've got the general idea. Board can answer any next-to-ness question for a given Point. Consider this alternative to your fragment:

  # determine if I am next to aPoint
  sub isNextTo {
      my $self = shift;
      my $aPoint = shift;
      my @neighbors = $self->{Board}->neighborsOf($self);
      return 0 != grep { $aPoint == $_ } @neighbors;
  }
[download]

Does that mean that the Board should be able to dip into the Point's inner workings and get its X and Y coordinates?

The word "Point" carries some baggage (but then so do some alternatives, like "Square"). Let's reframe the question by using "Cell".

Why should a Cell know its X and Y coordinates? Clearly, somebody needs to know them, but why Cell and not Board? (You might have good reasons to go one way and not the other, but this is a good question to ask yourself before deciding.)