I can't remember what the solution was, but it was pretty easy. I know there is already a thread explaining how to fix this. Use Search and type in something like Tire Smoke?
The alpha layer of your smoke file is faded beyond the edges of the file, so you are not getting a full fade to transparent. The square you are seeing is where the alpha fade meets the edge of the square file.
Take a look at the alpha layer. The white must not extend to the edge of the file at all. The outer edges must be fully #000000 or you`ll see some amount of squareness in your finished smoke.