I don't think that object size actually matters as you can have small cone with more polygons than big building and more polygons = slower performance. Of course amount and size of textures is affecting performance too.
One way to render a track efficiently is to break it into sections along the length of it. I.e., there might be 20 sectors from the start line back to the finish line. When you're in sector 5 you render the stuff in sectors 4 and 6 (ahead and behind you). This is very efficient for a sim like LFS where the circuits are all closed and there's no branching off. If this is how it works, I can see why it might not be possible to take a right on a road on sector 5. You don't ever proceed to sector 6 or 4...
I don't know how it works in LFS, but maybe it's something along those lines, which could make it not really possible.
That being said, I'd have a ball with such a "track" too. Would be fun