Hi, I'd like to help you with your question.
There could be many reasons why this man is not able to handle having a real relationship. He could have a fear of intimacy, be emotionally unable to express himself, have a secret such as being married, or he could just be very self centered and not care how others feel.
It sounds like this man is giving you all the signs that he is unable to maintain a relationship. For example, he only sees you for sex then wants you to leave right afterwards, he did not initiate contact with you after you left him, and he does not want to talk. The chances he will change are very poor unless he wants to change. And you also may want to consider the future in this relationship. If he is like this now, what would engagement or marriage be like with him?
If he really cared about the relationship, he would make an effort to get to know you better. He would talk to you, spend time with you outside of sex, and make an effort to advance the relationship. There would be no question on your part about how he feels about you.
I agree with you, this man does not respect you or your feelings. You deserve better and it is a good idea to move on. If you feel you are having trouble doing so, consider counseling for yourself. It would help you learn what attracts you to this type of relationship and how to address it so you can seek out healthier and happier relationships.
I hope this helps you,