Hello, I'd like to help you with your question.
It certainly sounds like your husband may have Asperger's. But it is just as equally possible that he has some pre set notions about women and he is taking those beliefs out on you.
The fact that he can say inappropriate comments to you and have to be told by a therapist that it is inappropriate behavior says a lot. His inability to bond to you as well, especially when you are vulnerable also speaks to his lack of insight and compassion. Although he seems able to show passion when he wants to with his causes and interests, it seems he is unable to do so when he is needed by you.
If he is able to be loving with your children and connect with them, then his issues sound more about his specific issues with women and connecting with others in general. His behavior in public is also of concern and it doesn't seem he cares about what others think of him even though he is a clergy person expected to uphold a certain standard.
At this point, the therapist you saw together should have provided some type of evaluation and diagnosis of your husband's issues. If they did
not, you may want to contact them to see what they have to say. Seeing someone face to face is vital in getting an accurate diagnosis.
Because there are several possibilities as to what might be going on with your husband (it's even possible to have overlapping diagnoses), you may want to focus on what you feel you need to do about your marriage. You do not deserve to be left out emotionally and be treated poorly by the one person who is supposed to be there for you. If you feel counseling was not helpful to address your husband's behavior (he has to be willing to change for that to happen), then you may want to consider going to therapy on your own. You need the support and someone to help you make a decision about your options in your marriage.
You may also want to decide if you feel a controlled separation might help. Although the idea of a divorce is difficult to consider, a controlled separation may help your husband see that you are serious about these issues. Here is a resource to help you:
Sometimes it takes a dramatic action to get a spouse's attention.
Also, get as much support around you as possible There is gap in your emotional support so having others to turn to can help, especially when you feel so alone.
I hope this has helped you,