Some times when people avoid others they are hurt, upset by something that has happened. There could be an issue that he needs to discuss with you, but is not willing to face it just yet.
It sounds too me that he wants to be with you and you had an affair with someone else. He knows how he feels about you and he is the one that feels he should be with you. Now what you have to decide is if you want to be with him.
I want you to real think about how you feel about him, how much do you love and care for him and if you see yourself with him in the future.
You also mentioned he has had family issues which could also be effecting him, but you need to tell him that you are there for him to help him get through any problems he might be having. It sounds like he is shutting off his feelings in order to deal with the problems he is having in his life.
What do you think I should do ? Actually this thing happened two years ago and I was not going on with anyone his friend was joking about it and he took it seriously.I was seriously in love with this guy but as I was unsure I moved on really quick. I have a boyfriend now. I was thinking I must have forgotten about my ex friend completely but I think I am just not over. I feel like going back to him but I cant as now he has a girlfriend and I don't want to come between them. I just looked up his profile on facebook and all his statuses were relating to me till 2011 and he kept it open to friends of friends. I just keep missing him more and more each day because all his feelings and emotions comes in front of me. I Feel so trapped and I feel regretful for not picking up his call. He even wrote a status that he misses me but he won't ever tell me because I am happy with my love :(. Life is so complicated. But thank you so much for your reply and removing all my doubts in a second :) .
Do you believe in the thought that feelings that come back are the feelings that never went? Will he remember me in future? I don't what to do :( I know I should move on but it so hard to do that