Exactly the same thing as with Harley Davidson motorcycles.
Drive one and you are instantly seen as a member of a motorcycle gang.
A few bad apples ruining the fun for the rest. But in the case of drifting it's a whole bunch on mindless dorks who ruin it for the few.
Only a few top drifters and millions of teenage dorks doing handbrake turns and power sliding.
The reason racing is not affected as well is the amount of media coverage it gets. You see racing on TV all the time but there is rarely a show about drifting on. So people have a pretty good hunch what racing is about. But drifting "knowledge" comes from the movies.
@K. Tsuchiya
Thank you for the drifting article. That was a good read.