The problem is abusive downrating (like making several accounts to downrate someone). We've already build a couple of tools to fight this, so it's getting easier to detect the abusers and harder to get away with fake votes.