I must say 'good for you!' in recognizing that this man's treatment of you is less than ideal and could very well be disrespectful and that of using you.
In life we must accept our own responsibility for how others treat us. We have to stand up for ourselves and insist on respect, as well as earning it. When we have earned the right to be respected and someone still refuses to honor that, it's only our problem if we continue to have that person in our lives.
You need to stand up to Mr. 'I have issues' and tell him that oh poor baby, we ALL have issues and we don't make others pay for them, suffer the consequences of them or somehow get the blame for them. If he wants to continue to see you it will have to be on a dating basis with activities you can both enjoy while getting to know each other.
If he's not up for that, well, goodbye, good luck and perhaps when he grows up and man's up he can try to call on you again; however, chances are rather slim that someone as good and desirable as you will have found someone happy to treat you the way a lady deserves to be treated.
I know you like him, but ask yourself why you would put up with this shoddy treatment and is this the guideline you want to establish in any relationship?
It will only get worse - the more you enable him to have such control and to treat you poorly.
You deserve better. So set your own ground rules and if he's truly interested, he will step up. Otherwise, I promise you, you aren't losing anything 'real' by letting him go