So if someone else knows my private key and voted for me not the same way i intended to vote, this will be visible in final results. And then you can start asking questions. If they have my private key and voted the way i intended, then its no problem :)
Why would others trust you to be telling the truth and not just lying to attempt to cause mistrust in the election?
Naive solutions have naive problems.