Err, you'd obviously cull irrelevant geometry when rendering or doing physics, but the data needs to be persistent. You can't just drop it once you've driven past. The next lap the damage you did to the track needs to still be there.
Well, that's what the vertices would be if you went the way of Sega Rally. But you can't spread them out very far because then you wouldn't get defined tyre tracks and other high frequency detail. Having a bump every other meter isn't good enough.