We receive much of our self esteem from our families and our relationship with our parents. Women receive so much of their self value through their fathers, and if that relationship was damaged, many grown women pursue relationships with men who are aloof, distant and hurtful...much like their own fathers. It becomes a cycle.
To break the cycle:
If you are able, continue to look at what you have told me about your own family. I hope you can now see that you feel the way you do about this abusive man because you are still looking to be accepted. You do not want to reject him because you have felt the same way in your own life. But, this is different. His feelings are not the same as your own, even if you think they might be. He is a different person, and you cannot and should not project onto him, feelings that are yours from your past. Therefore, when you set limits on what you will accept from him, yes, he will feel badly, but it is not the same rejection and feelings that you experienced. It is different. So, you can feel free doing what you need to protect yourself as he will not feel the same pain that you did as a person growing up in your family.
I would like to recommend a book to you. It is Adult Children of Abusive Parents by Steven Farmer. It is very inexpensive on amazon.com and other internet sites. It outlines exactly what you have experienced and shows you how you may react to the pain of this experience by choosing damaging love relationships.
Mindy, you have taken some good steps here. Look at what I wrote about separating your own feelings and past from what he has done to you. They are different. When you see that he does not feel the same things as you do... and that you were victimized by others and that he is an abuser, not a victim...you will find the strength to stand up for yourself, without apology and will be able to set limits on this harmful relationship. Steven