I know you think this is about him... but consider this: Guilt is when we do something that we are not proud of. Shame is when we do something we are not proud of and someone else saw it or knows about it.
Granted he says horrible thing to you... but I suspect that if YOU really got okay with all that happened, his comments and his attitude would have a lot less effect on you -- he's pushing buttons in you that already exist.
Deal with your guilt and shame and you'll find the courage to do what is right by all of you. Confront (in you) what happened and you'll be in a more powerful position, emotionally.
Keep me in mind if you ever want to chat!
I wish you the best of luck... best wishes, Nancy