Howold are the two of you?
How long did you know him before you moved with him?
Where did you meet?
I know it hurts. The problem is, if one person tells you something, then you may think its possible its a fluke or it may not be true, but when more people tell you the same thing, then you know there may be some truth to it. The best indicator of future behavior is past behavior.....the fact that his other girlfriends complained about it as well means that it's not personal towards you, but a problem he's had for a long time.
If he has never seen a therapist or had counseling, this might be a good time for him to think about it. Perhaps if he knows how badly this is affecting you he might consider it. There are many men (and women) who are not affectionate, whether its because of the way they were raised or because they have been hurt, but if someone truly cares about you they should be willing to compromise.
If he is willing, perhaps you can try some touching exercises. Hand massages, foot massages, full body massages (that don't necessarily lead to sex) are all good ways to help him get used to touching and being touched. Ask him if he can remember to hug you or touch you at least twice a day. Once he starts doing it, it can become more habitual and he may get over his fears or whatever it is that's stopping him. If he can't remember, he can put two colored rubber bands on his wrist, and when he sees the rubber bands, it can remind him, or you can put a note on the fridge "don't forget my hugs" and this can help him remember. I know it might seem silly, but when it works, it might not seem so. :)
I welcome your thoughts, let me know if you want to talk more.
You should not in any way feel rejected, remember that you're not the first person to tell him this. You also should not feel afraid, this is your man, the person you've been with for almost a year....if you can't feel comfortable talking to him then there is more of a problem than just affection. Just let him know that you want to try a few things to help him with his affection issues. As for the massages, don't make it optional. Pick a day of the week, ie; Friday nights where it is massage night. Light candles, play some music, and have a glass of wine or champagne (whatever you like) lay a blanket on the floor or set up the bed so that you're both comfortable. Each of you gets 15-30 minutes, then you switch and the other gets a turn. You can also take a natural bristle hairbrush and brush in strokes down each others bodies. Don't forget the hand and feet massages as well. I know you're nervous, but it's worth a try.