Nobody voted on your node in the first 30 minutes. Within 40 minutes, it got one upvote to +1. Within 1 hour of posting your question went to +2. After that it moved down to 0 and then back up to +3 (after 2.5 hours). After more than 7 hours, it finally made it to -3. This was all due to 17 people voting on your node over the course of those hours.

The most likely explanation is that you are simply exaggerating both the times and the reputations involved.

As for why you got a handful of downvotes? Because a handful of people disliked your question enough to downvote it.