You have to be sure of what you want before you end this. A good way to make this judgment call is to try marriage counseling so you can explore this with each other in a safe environment. There may be more that you don't know that would be a positive influence on the relationship. If you need to know why this is it gives you a place to ask this. There is then a way to talk about this freely. If you think that it is pointless then consult a lawyer. I would consider marriage counseling since the relationship has been good up until this point. If you have a good relationship you can work to address the point that aren't so good. There has to be good points since it was going so well. Also discuss with him why he said this. Maybe he was joking.
If this was helpful press accept