I don't think you are blowing anything out of proportion. I didn't realize that you were together at the time. You have every right to be upset. He should at the time have found another solution than staying in the room. I don't blame you for having difficulty with this. If nothing else it is hurtful.
As far as ruining the evening, you did the best you could. You were upset and didn't cause a scene. You merely removed yourself from the discussion and collected yourself by walking away. You are allowed to have your own reaction. Instead of him defending you, he discusses your being a bitch. I would be hurt about that too. This can only be resolved between the two of you and him having a genuine discussion with you without insults and finding his sense of empathy. Don't let go until it is resolved. You can also ask to speak to someone if you need intervention