Are you still with the same man?
If not, how long were you with him?
How do you know he lacked attention from his mom?
Sometimes people do carry what happened to them, when they were small or in other relationships, into their current relationships. This is called 'baggage' and it can usually cause problems in the relationship. Either the person doesn't realize that they are carrying these things, or sometimes they do realize it, but they can't help it anyway.
Sometimes the ugliest people can be very beautiful on the inside, and sometimes the most handsome or beautiful people can be very ugly on the inside. I noticed you've mentioned ugly a few times, here and in your other questions. We can't always judge people by the way they look, some ugly man could possibly make the best mate, or some ugly girl could make the best of friends, you never know, you have to try and look beyond that.
It could be he cared about you but was afraid of being used for his money. Maybe someone in his past used him for his money and did not care about him.