Through reading posts and old posts i have seen it come up. A quick search found this thread. There's plenty others as a simple search will show. And there's always a reply saying to ignore them and they will go away or the like.

I don't see how the thread you linked supports your earlier assertions that eliminating downvotes is "oft discussed" or that there is no consensus that they should remain available. Searching the text of that thread for the string "vote" turns up exactly one mention of downvotes (by tye) and it is hardly critical of their existence, given that he's describing a situation in which he himself will issue them:

Then there are the people feeding the troll feeders. I've down-voted a ton of troll fodder and almost always I find that no small number of monks have already up-voted it. So it is no wonder so many keep dishing out the troll chow.

Searching the text for "voting" adds a mention by Argel that he didn't want to name others without also naming himself, giving "retaliatory downvoting" as an example for why he wouldn't want to do that, but this did not lead to a discussion of the topic (the node in which he mentioned it received no replies) and he did not in any way criticize the existence of downvotes, only acknowledge the possibility of their misuse.