I misread the code when I wrote the below. Now, I'm unsure why MANIFEST.SKIP isn't being used. Nonetheless, my above post stands.
Module::Build::distcheck() in a round about way has this line: $self->_check_manifest_skip unless $self->invoked_action eq 'distclean';. That means the MANIFEST.SKIP file is not being used when the distcheck command is run.
Add MANIFEST to the MANIFEST.SKIP as hippo suggested, and just let ./Build test be enough... it verifies whether MANIFEST is up-to-date with the t/manifest.t file anyway.
