But there's still the chance someone you didn't like/wanted to avoid/owed gambling debts to would join a server and find you anyway.
Unless, of course, you mean people in general and not specific people, which means you'd be alone in a server. If that's the case, maybe you should race offline when you're in the mood to be left alone. That, or start your own private server and not share the password with anyone.