If he's never cheated that you know of, then you can't accuse him of it without proof. I believe that all women have what's called 'women's intuition', and if you feel strongly that there is something wrong or that he's hiding something, then chances are he probably is. You have to question yourself, and ask yourself, are you normally a paranoid person? Are you distrustful by nature? If so, then it may just be that you are normally suspicious.
If he's using other things as an excuse when you try and talk to him, then he may be in denial. You have to let him know, even if it means writing things down and addressing them one by one, that when you discuss you and his issues that you don't want him bringing other people into it, as this is your marriage and your relationship and no one else's.
The lack of intimacy is an issue as well. Are you making 'alone' time for the two of you. A time where they two of you can be alone and not get interrupted by kids, family or friends? This alone time may not always be used for sex, but also for hugging, cuddling, talking, and generally getting to know each other better.
I welcome your thoughts, let me know if you want to talk more