There are a couple of issues here. He is 50, and has never married. That being the case, for years his most common way of getting sexual satisfaction was through masturbation, and may not, in fact, be capable of sexual intimacy with you, or be greatly afraid of 'failing' in some way.
Secondly, it's time for some home truths - he needs to understand quite clearly how this is making you feel - unloved, unwanted, rejected I suspect, and also that you are not prepared to go through the rest of your life like that. I suggest that you tell him that unless he agrees to counselling with a sex therapist (possibly as a couple), you will have to consider what your future is going to be. If he not given reason to co-operate and change, he will not. He's getting what he wants already.
It's all very well to say that he loves you, but are his actions in tune with his words? In your shows, I would take this VERY slowly, do what I have suggested, and see what transpires over the next few weeks