It's not the slowness that annoys me as such (although there is a noticeable framerate drop in LFS and other games), it's the entire bloated layout of the OS.
If I have a bunch of documents in a folder, and then a media file, the last thing I need is for the OS to 'recognise' the folder as a media folder and give me something stupid like a thumbnail view or whatever. When I say apply my details layout to every folder, I mean it. I don't mean apply it some, then piss about with others. I just don't have time to waste messing about with a useless layout.
I used it for about a year, then just gave up. Of course I realise we all have to learn new operating systems given time, but learning the upgrade from OS X Tiger to Leopard took all of about 5 minutes, and every new thing I found was actually useful. Similarly every time I upgrade Linux things just get easier. I still wouldn't recommend it to the average user (don't know about these 'user-friendly' distros like Ubuntu, didn't like it myself). However if you need a job doing, get the right tool for it first.
As a multi-OS 'power' user I now rarely use Windows for anything other than games, thus my streamlined XP does the job perfectly for me.