I gather a lot of people are dropping vBulletin for XenForo, led by former developers of vB.
However if your looking at building a custom solution then alternatives on the market arent going to make all the much difference.
We're toying with the idea of building our own forum, personally i think its just too big a task, especially as it really needs to be functioning when the site goes live, even if its just the most basic of features, coupled with everything else required to go live its a heck of a lot to do.
Im not really sure there are enough benefits to it though, but then i wouldnt be the one building it and im not sure what the guy suggesting it has up his sleeve already, we already have a solid core to build from before really starting which i wasnt expecting to have, so maybe thats half the battle here too. Dunno.
I just get the impression people will miss a feature and moan about why you've not gone with vB 'cos its better than the &$*£ running at the moment' etc, and despite feeling like its quite an achievement what you've done, people dont really care and they'd settle for vB for all it matters to them. If theres something you can achieve which vB or similar cant offer, and your willing to put the time & effort into it all, I suspect it'll feel real good actually pulling it off.