Hi, I'd like to help you with your question.
There is nothing wrong with you. There is, however, a conflict between you and your husband about how to get your needs met with each other.
Let's take your husband first. He has unresolved issues from his childhood. He was abandoned by parents who both had issues that took them away. So he has needs he never had met. His anger, not standing up for himself and lashing out are all signs that he feels empty somewhere inside. Being a victim is what he understands. He doesn't know how to meet his own needs.
Counseling would help him. But he may be afraid to confront what is really bothering him, which is most likely the pain of being abandoned and maybe abused. So he carries it around with him taking it out where he can (anger) and trying most of the time to be what he really is inside, a good husband and father but not overcoming the pain from his past.
On your part, you are coping with someone who feels incomplete inside. You want the same things your husband does, a good stable home and happiness together. But coping with his issues have left you feeling unfulfilled and looking for a way to get what you need. Trying to get what you need from him has not worked so you looked elsewhere for a while. But you returned to your marriage, I believe, because you felt what you needed is still there. You may also have some unmet needs from your past that cause you to feel this way.
You are trying to work on your marriage, but at this point it sounds like you are doing this alone. You both need to come together on this. Go with your husband to counseling. Along with dealing with your marriage issues, your husband also needs screened for Aspbergers. If he does have Aspberger's, that opens up a whole new way of dealing with your situation.
But if he does not have Aspberger's, marriage counseling will help you both learn better ways to communicate so you can resolve issues and you can get what you need from each other. It will also help your husband address his past so he can heal and move on. This will help your marriage improve as well.
Also, share the materials you have been studying to improve your marriage with your husband. You both need to discuss what you learn and share better ways to help your marriage. Try a marriage retreat as well. It can help you bond so you can get a better start to repairing your marriage. Here is a link to help you: http://www.familylife.com/site/c.dnJHKLNnFoG/b.5846045/k.F479/Attend_a_conference.htm.
I hope this has helped you,