How long have you been seeing him?
How old are the two of you?
Has he said why you aren't meant to be together?
Are you intimate with him?
I'm of the opinion that no one is too busy to make a simple phone call. Even if they call you on break, lunch or when they go to bed, but that's just me. If he feels he can go for days and there's no way that he can take a second to call, then you have to decide if that's good enough for you. One of the biggest mistakes people make is thinking that they can be the one to 'change' the other person, but the best indicator of future behavior bar none, is past behavior. If this is the way he's been for two years and he's not given any indication that he would be willing to change for you, it doesn't mean he doesn't care for you, it just means that what he does and how he does it, works for him, so why should he change?
What you will need to do is to decide if this is working for you. If it's not working for you, then you will need to tell him what you want, and see if he will compromise. I welcome your thoughts, let me know if you want to talk more
Its hard to say without hearing him say it, so that I could hear the inflection. If he says it in a teasing manner, then it doesn't really mean anything but what he's saying. It could be an excuse, his way of blaming you for the intimacy, and not taking responsibility or it himself, which is what enables him to keep his distance. Like you are seducing him against his well.
If you're going to leave him alone, it should be because he's not willing to call, or commit to you. You need to have someone who is willing to love you the way you need to be loved, not just when they feel like it.