To be honest, I wouldn't worry about it that much.

I for one never even see CPAN ratings. My interaction with CPAN is all done on the command line.

If you want people like me to consider your module, the things you should be worrying about are, most important first:

1. The name. If I can't guess what your module does from its name, I probably won't even glance at it. (Cute names like "Moose" can work, but only if you put the effort into publicity.)
2. The README. This is the first thing I look at once I've found a promising name. It must contain a clear statement of what the module has to offer. If it doesn't catch my interest, I move on.
3. The license. There are some nasty surprises on CPAN -- complete modules that can only be used for non-profit purposes, for example. I need to be able to find the license quickly, and it needs to be a standard one -- preferably "same terms as Perl itself".
4. The POD. If the README looks promising, I read the module's documentation. I'm looking particularly for clear usage examples.
5. If the documentation doesn't quite convince me that I've found a quality module, but it looks like it might do what I need, I look at the code itself to see whether it looks well written.

Arbitrary scores from random reviewers don't even figure. I probably don't know those people, and I have no way of judging whether they're qualified to review your code. Why would I care whether they gave it one star or five? Forget Web 2.0-y fluff; it's the meat of the module that matters.