libxml2 supports RelaxNG, and XML::LibXML exposes part of its API (see XML::LibXML::RelaxNG, so you might want to look into it. If you find that you need more of the API, a little proding of the maintainer would probably get you what you want (sending patches would work even better ;--).

I don't know about extending the notation though.