You need to trust your instincts for heaven's sake and change your number; avoid this guy like the plague that he is.
--- First, I suspect that he was afraid you were leaving when you went to the ladies room that first night.
Second, I strongly suspect that his reaction was to 'dump you' before you had the chance to dump him.
The biggest mistake was seeing him again. Stop this dysfunctional relationship before it goes so far as to truly damage you.
This guy has started emotionally and verbally abusing you within hours of your first date. It will only escalate over time. He'll apologize, he'll give you things, make you promises, offer some of the best excuses and rationalizations you will ever hear - and then he'll do it again.
You can also expect that if he thinks you're fooling around on him, he'll cheat on you and then blame you for making him think you were seeing someone else.
Nothing will ever be his fault. This has all the earmarks of something that may escalate into physical abuse.
Surely you value yourself more than this. You deserve a good man, not damaged goods. You cannot 'fix' him, your love won't make him better and cure him of his mistrust because his heart has been broken (have you heard this already?) and of course he has "walls built up" and he won't let anyone in.
You can only be responsible for yourself. Run, do not walk, away from this one.
There is much better out there.