It would be even more annoying to be unable to remove from the grid someone who'd be AFK (or asleep) and preventing the race to start by not clicking on Ready.
BTW, it reminds me of a problem I've seen the last time I was on B&J: when the track changed, a couple of guys clicked Join and Ready before everyone else, and the race started. They voted to restart, but it was too early and the system didn't allow it. After a moment they pitted (or where spectacted for crashing and going the wrong way), and then we were all stuck in spectate mode with nobody on track. We decided that everybody had to leave the server and come back, to force the race to end, but after several tries we found that one of us was AFK (or asleep) and didn't leave. And so we were stuck.
I don't know how it ended, because I left when I realized that we were stuck until he came back or was finally kicked by X-System, but I guess everybody just ended up doing the same. Maybe in the next version, X-System could work around this situation by prompting people who've been in spectate mode for some time with a dialog, and kicking those who don't answer this dialog, or something?