100mbps is pretty common where I live (and that is in an area just north of stockholm, sweden). But I still get this problem sometimes and I think that is because my SSD is not good at writing stuff... :-(
When connecting to the server I get a lot of skins (as I guess you do) that has to be written to the HD (SSD). But if I try again it usually works.
So, do you have a SSD or a regular HD?