It's a problem of determining when the opinion of a minority should outweigh the feeling of the majority that makes this difficult. Restarting a race, in the case of T1 hell, I'd accept the top 3 voting as authoritative and allow the restart early, but in the case of a vote to end it's a much more complex process to determine "fairness" when accepting or denying a "minority" vote. Anyone in 4th but on a crackin' lap, for example, could be denied their lap record.. a lame example, but valid. Majority rule is the *most* fair of all possibilities, IMO, on vote to end.
Perhaps only allowing votes to end after the "race restarting" timer is triggered would be the fairest. It's a small window, but should reduce the risk of misfortune in the vote.