It should also be noted that AUTOLOAD introduces something like a 400% performance penalty over normal method calls

That penalty can be avoided for every but the first call to the autoloaded setter/getter function, if the AUTOLOAD block generates a subroutine living in a typeglob, just as AutoLoader does:

sub AUTOLOAD {
    my $self = shift;
    my $type = ref($self)
        or croak "$self is not an object";

    my $name = $AUTOLOAD;
    $name =~ s/.*://;   # strip fully-qualified portion

    unless (exists $self->{_permitted}->{$name} ) {
        croak "Can't access `$name' field in class $type";
    }
    my $sub = sub {
        my ($self,$value) = @_;
        $self->{$name} = $value if defined $value;
        $self->{$name};
    };
    {
        no strict 'refs';
        *{$name} = $sub;
    }
    goto $sub;
}
[download]

Each subsequent call to the method accesses the sub normally, i.e. through typeglob lookup.

not to mention the fact that your code is broken once you introduce inheritance since you have no mechanism to inherit the fields. Sure that problem is solve-able but not without a bunch of tedious plumbing as well.

That plumbing amounts to writing a proper import() subroutine which calls AUTOLOAD for the inherited fields and exports them to the inheriting class. But then, you're right, why reinvent the wheel (there are reasons, though) if there are plenty modules out there which handle that. There's Moose, classes, Class::Accessor, ...