Is he insisting that you get married?
Any other relationship problems?
Im a little confused because you're saying "we" but say that you think you are harming him?
I'm getting from you that everything is good with the relationship, but you want to travel and do some nice things while you still can, but he's not in a position to do the same. Whether its because he refuses to spend money, leave his daughter or is not feeling like its a good time for him.
He shouldn't get mad at you for saying that out of this world sex is great for 50, because there are so many at that age that have trouble in that department. It is great when you can find someone who you have sexual as well as emotional energy with, regardless of age.
There's nothing wrong with staying in this relationship if you love him and you're both happy. There's nothing that says that a relationship needs to be a particular "way", you are the only one who needs to make a decision about how your life is, your life is what you make it. Period. If you feel that the two of you should make more of a commitment to one another, you could always have a gathering with closest friends and just have a commitment ceremony. If you feel that you need to take a break and do some things for yourself then that's ok too. It's really all about what you need right now. He's already doing what he thinks he needs to do, and you are entitled to decide whats best for you.
I'm a little unsure if I've hit upon what you are asking, but I certainly welcome your thoughts. Let me know if you want to talk more.