You have to decide if this relationship is worth giving up because of this issue. I totally sympathize with you. I would be losing it too. It is very difficult when you think your partner is eyeing up another woman. He has given you a valid reason though as to whether they are fooling around so now you have to work on him respecting you. Instead of saying are you having an affair start small. Tell him that you have noticed that he watches her and it makes you uncomfortable. Explain to him how you feel each time she invites him over and why you don't want to go. Don't say are you having an affair. That suggests that they are already fooling around. You have to find a time where he is not annoyed and tell him that it is more then that. If you can get across that the whole situation makes you uncomfortable then the rest of your questions will be answered. But don't blame.
There should be a compromise to everything. However if he is that defensive then you take the chance that he won't compromise. If that is the case then you run the risk of him wanting to maintain this relationship at the risk of ruining yours. There is a chance this will happen and then you can't talk to him about this and you have to make a decision.
If this has been helpful press accept