what stops an intruder from changing your e-mail address twice (...)

Nothing, but you would still receive the notification for the first change. If the old (previous or original) address is invalid, the mail gets bounced - bad luck (or doesn't matter in the more likely case you wanted to change the address because of that).

I know it's a weak proof, thats why I put it in quotes, but I think the notification might be a good feature.