Now I know what causes this "distortion". When I switch the car I spectate, it sometimes goes buggy. When I start to spectate someone else, it goes away.
However, sometimes it starts in the middle of a race when I drive, so it isn't always about spec...