A clear definition should be made between being a couple and being just friends. Some times men rush into the physical side of the relationship and then you end up losing a friend (and if the relationship does not work out end up losing a boyfriend too)
If you have noticed that he had called the other women in the past, you need to find out why he is not calling you. Anyone can claim love but love is also expressed in actions not just words. Perhaps you ought to test him. No one who loves another and respects them would accuse them not being affectionate but rather try to help them understand how to make the relationship better and make things work out. It takes two to keep a relationship going so you ought not to feel guilty about what he had said to you. That is probably his excuse for his own lack of effort to make the relationship strong.