It appears you are responding to an imaginary complaint about downvoting. My complaint is about people who do not post corrections to others incorrect posts, instead they only downvote. Downvoting somebody for incorrect information does not help the community, b/c no one else can benefit from it, unless you are sorting by rep which as you've said 2/3 of the visitors can't do.
So downvote randomly if you feel like it, I don't care. But if you are going to downvote a node due to incorrect information, please post a comment.