one could gain sainthood just by logging in and downvoting anything one sees.

So?

Someone wastes x amount of their time, doesn't learn anything, and the only effect it has is that nodes that they immediately see (which may be on the frontpage and if that's the case are more than likely highly overvalued anyways) will have their rep reduced by one. Doesn't seem like a big problem to me.

I really think one should not receive any XP when downvoting.

I disagree. Then virtually nobody would downvote, which would create problems of its own. If you're going to remove XP bonuses for downvoting, it should be done for upvoting as well, but that's another debate (and not a very important one IMHO).

Update: To quote Mark Twain:

Whenever you find yourself on the side of the majority, it's time to pause and reflect.

While I'm still pretty sure removing the xp bonus for downvoting wouldn't improve the moderation system at all, are there other additions (or removals of current features) that would?