A deed is a legally dispositive instrument. If you have a deed for (example) Lot #11, but your original contract stated Lot #12, then you own Lot #11, and it would be up to the hypothetical purchaser of Lot #11 to prove that they had a contract for your lot and you are unjustly enriched at the purchaser's expense. The same theory would apply to the seller, i.e., the mistake must "unjustly" to your benefit. Otherwise, the deed cannot be declared void.
Proving unjust enrichment
, usually requires some sort of "scienter" (deceitful intent) on the part of the person enriched. Nothing in your facts suggests anything other than a simple mistake, either in the contract or in the deed. But, if you've been paying for the property for several years now, this suggests that neither the seller nor any hypothetical or actual purchaser has any idea that there is a problem. Most likely they never will -- but, if they do, you would have a defense called "laches," which means that the plaintiff has unreasonably delayed in pursuing its rights, and you have been unfairly prejudiced as a result. Paying taxes on the property demonstrates a prejudice. Constructing a building on the property would be enormously prejudicial, because of the value that you would have put into the property.
BotXXXXX XXXXXne, you are probably better off building than not, because it improves your legal position in the event of a lawsuit. That lawsuit may never come -- probably should have occurred by now.
The only way that you could completely remove the risk would be to file a "quiet title" suit and name the seller and the current and prior owners of the other lot as defendants, so as to obtain a court decree that the property is yours. That will cost you a lot of money (est. $20,000 in attorney's fees). That's the risk-reward ratio here. You'll have to decide whether or not it's worth it to "let sleeping dogs lie," or wake them up and hope you don't get bitten!
Hope this helps.