I've messed up my side mirrors on occasions. Love that bendy glass.
It doesnt break yet I guess because the Devs are working on car physics before working on how well things break. If you've smashed your car so hard it has a broken rear view mirror, your race is probably over. I imagine they are working on making things good for those still in the race and those crashing into walls get eye candy stuff later on.