He is stuck in the past. His behavior is not indicative that he is 100% ready to move on or accept some responsibility for his actions. If you do not have any other intimacy besides intercourse, he is not focusing on your needs. Saying comments about your affair or threatening to have one himself are harming the marriage.
While you're doing your part to fix things in the marriage, he is undoing your work through his behavior and words.
Find out what is it that he wants from you and the relationship as a whole. Is he happy/satisfied with the way things are. You're not satisfied so it would be unlikely that he is satisfied. He would have to move on and you will want to find out if he's ready and willing to leave the past behind. But, you can not do this for him. He has to step up and do his part. Find out what would be realistic for him and insist that he does it. Don't blame him but you can still confront him. Point out what you've been doing to make things work out and ask him to do the same.