Is this someone you're dating or thinking of dating?
How old are the two of you?
What kind of issues does he have?
How was his relationship with his mother?
The more info you give me, the better I can help
You know what's interesting is that it may not be just about his mother. His lack of affection could have to do with a wide variety of different things, including but not limited to
Sometimes it can be some type of mixture of all of these things that created the way he is. It's really hard to tell for sure. The good thing is that it hasn't affected his sex life. Perhaps you can try a few touching exercises that may help him become more comfortable with touching. If he is an enthusiastic lover, perhaps he would be willing to try some new things. See here a great article on Tantric sex, consider giving each other full body massages, brushing each others bodies with a natural hair bristle brush, foot and hand massages, washing each others hair....all things he may reject or feel uncomfortable with at first. Keep in mind that none of these things must lead to sex. You should in fact do many of them without it leading to sex so that he can learn that touching doesn't always have to lead to sex, but can be just as fulfilling.
I welcome your thoughts, let me know i you want to talk more