NO one can tell you what to do in this situation. Only you can make that decision. However this is what I see going on here.
1. He is not available to you, emotionally or physically. The distance makes it hard to visit, and his relationship with his wife creates energy that is not available to you.
2. The energy you have with him, detracts from what he is able to give his wife. So his wife is being short changed in the relationship, because the husband is truly not 100 percent there.
3. If he is focused totally on his wife and she eventually passes away, or heals, then latter he may be more available to you. (if he leaves his wife).
For these reasons, while it his hard to do, if you were to end it with him, you free him up to be devoted to his wife. He may learn to appreciate that later on, especially if his wife gets bad and passes away. If she is not terminal and recovers, then he may also appreciate it.
Taking care of a sick spouse can be drag. It is very hard. For now you are a distraction from that. BUT when it gets down to it, it is not fair to him, his wife or you.
YOu deserve someone who is available to you full time.