Ok, I make a test. I try to copy your scene only to make a test, (It`s yours).
10 min and 30 sec.
296.769 polys
Why do you have 3 millions?
Real bricks?
It´s a differents texture yes, but I think that`s not a BIG different.. ( I use Bump in the texture)