I wonder what everyone uses in their real cars. The only time I've ever driven with shoes on was when I took my driving test. I only wear thongs/slaps/flip-flops (take your pick) or vans slip-on sneakers, so it's practical for me. Socks are very consistent, grip-wise; bare feet can get sweaty and slip off most pedals. Fortunately, socks still allow me to put the pedals between my toes for ultimate fine control and feel of the vibrations.
I liken driving with shoes on your feet to driving with your elbows on the wheel and shifting with shoes on your hands--at least in relative terms of control. You wouldn't care so much about smoothness when racing though, nevertheless, I don't see how it would hurt going barefoot/sockfoot (unless of course you're scared of your feet getting niki lauda'd).