No you are definitely not off base here. Your gut is telling you the right things. Your husband refuses to go to counseling because he is not willing to own up to this being his fault. Many people who chronically cheat also have issues with being Narcissists. They feel that they are right in what they are doing because they need it. It sounds like this may describe your husband.
He is telling you explanations that he feels you will accept. If he really had issues, he would include you in resolving them. You would not be feeling left out and you would not be questioning yourself.
There are numerous resources that will help you sort this out. The more you know, the better you can address what you are dealing with.
The first one I recommend is counseling. Finding a counselor that is knowledgeable in affairs and narcissism would be ideal. And go yourself.
Two, learn what you can about infidelity and narcissism. Also, you may want to learn about emotional abuse. It sounds like there are at least traits of abuse in your relationship.
Infidelity: A Survival Guide by Don-David Lusterman
Surviving Infidelity: Making Decisions, Recovering from the Pain, 3rd Edition by Rona B. Subotnik and Gloria Harris
Should I Stay Or Go? : How Controlled Separation (CS) Can Save Your Marriage by Lee Raffel
Why Does He Do That?: Inside the Minds of Angry and Controlling Men by Lundy
The Wizard of Oz and Other Narcissists: Coping with the One-Way Relationship in Work, Love, and Family by Eleanor D. Payson
You can find the books on Amazon.com or your local library may have them for you.