I disagree with any kind of penalty for downvoting. If you're doing it to remove trolling, or a simply dangerous suggestion, then I really don't think you should suffer for doing something to help give people consider'ing the node a clue as to what people think. If people felt that downvoting was more frowned upon than it already is then I don't think they'd do it, thus losing us a valuable ability and indicator as to what people think.

People who downvote with restraint and discretion are doing Perlmonks at least as much good as someone who upvotes wildly on anything they see. I don't see why they shouldn't get at least as much XP-recognition in that.

Myself I try not to downvote unless I think it's actually harming Perlmonks and the people who'd take this advise to have it there, or if it's a question which shows absolutely zero effort before asking.

It's not censorship, it's not a personality campaign, it's quality control.