I agree. You are in a very confusing and tough situation. What we really need to do is piece this by parts.
First off...you and your husband This is probably what led you to this other man in the first place. It is very strange that your husband will not have sex with you in over 3 years. That is not fair to you at all. If he forgives you for what you have done in the past and he himself has done it as well then he needs to act as a husband. If it is too much for him to deal with and is unable to have a sexual relationship with you then he should be honest and it may be better for the two of you to go your separate ways. 3 years is a long time. The question here is do you want to save your marriage? If you really do then I think you should drop this relationship with this other man. I understand that people cheat for various reasons. Me personally am against cheating for any reason. I feel the best thing is to deal with the problems in the relationship first. However, I respect people who do cheat and know that there are a lot of people as yourself who are not cheating just to cheat, but have much deeper reasons and need genuine help.
I think first you really need to see if you want to stay in your marriage and think how would you feel if he was doing this. It would be good to try and get him to go to counseling to resolve this problem. However, if you feel you are done with your marriage and do not see it getting worked out then the best thing for you to do is leave. It really depends on where your heart is in the marriage. I understand you must feel like friends with your husband and this is leading you to this other man and I agree with you, but the truth of the matter is that you are married and this should be dealt with first. Keeping your husband without any hopes of rekindling the marriage will waste both of your time. It is best to decide where you want to go with this first.
2nd part...This other man....I do not see much hope here. I still do feel that he is comfortable with you since you are married as well. If you do not mind having an affair being married and him as well then it is your choice to continue this, but you have to know that this is temporary pleasure. I do not see this as something that can grow over time. Well sure it can grow. The two of you may have the best of times, but his wife by what he says seems to always be in the picture. He definitely has an issue because she does not want to move, but despite that he still does not want a divorce, so does not look like that will happen anytime in the near future.
With this man you will always be the 2nd one. I do not feel this is a healthy relationship for you. This will just lead you heart broken since it will never be the way you really want it to.
If you are happy with that then you can continue it, but remember that it is what it is. You should not look for more because then you are setting yourself up to be hurt in the future. You just need to enjoy it for what it is.
However, the best advice I could give you is to try and stop seeing this man and decide if you want to be with your husband or not because then you can focus all of your energy in trying to make it work and if you do decide to not be with you husband then you can find someone that will be 100% devoted to you and not to another woman. That is what you deserve. Don't settle to be 2nd when you can be first.
See, that is my true opinion, but I understand that there are some that enjoy being married and having an affair on the side. If this is what you prefer than you need to realize that it pretty much will always be a confusing situation. You would just need to enjoy the time you have with him and not worry about his feelings towards his wife.The issue he has about the hometown and etc is meaningless since he still chooses to be with her. That is something he needs to work on himself.
I hope this was helpful and brought some insight to the situation