Hello and thank you for your question.
Your husband's behavior sounds inappropriate. You are allowed to decline sex if you are not feeling in the mood. It sounds like you are only doing this on occasion. His reaction of pouting and sulking is immature. His threat of having an affair, due to this and then placing the blame on you is manipulative. It's quite possible that he is thinking of having an affair and needs some way to justify it, or is simply being controlling and trying to scare you into doing what he wants. Giving you the silent treatment and refusing to do anything with you, when he is mad about this is destructive and an over-the-top reaction. It sounds like from your behavior, he knows that you are scared and will try hard to please him. Instead of him appreciating you or trying to reach a mutual solution, he is more concerned with just himself. I would stop bending over backwards to get him to talk with you. Call his bluff...tell him, if you decide to have an affair, that will only be your choice. You are an adult responsible for your own choices. If you choose a destructive one, I cannot promise you that I will stick with you. Don't let him control you. Also, you two need marital counseling. If he refuses, again tell him, that if he is not willing to get help with these issues of how he behaves when you say no to sex, than that is his choice. Tell him if he refuses to accept your apologies, that is his problem, not yours. Then go about your business....and let him see that you are not going to do whatever he says, whenever he says it. Also, I highly suggest that you seek some counseling, even on your own(if he will not attend) as you could use some support with this situation. Don't let him manipulate you. Please click ACCEPT, if satisfied. If not, please REPLY to me. Thank you.