He states he takes you seriously, that you are his girlfriend but he also acknowledges he was before with another woman who considered him as her boyfriend while he didn't take her seriously. If we take that concrete situation into account , plus the other behaviors you have identified that seem to be red flags here, then it could become a concerning situation.
It's interesting to see how many people could use children as good reasons for keeping new relationships totally aside from their real whole life, from integrating them in their realities. it is not uncommon to find people who literally alienate new girlfriends or even partners to the point of keeping the relationships so secret, that the other person has not way not to feel isolated and nos as a real part of their boyfriend's reality.
Many times I do not see how being fully honest and open about having a serious relationship could "damage" children, 4 years a marriage has ended. There could be multiple reasons people in his shoes could use, but most times would be codependent excuses and not assertive points justifying such behavior. It is very common to find people in your shoes to find themselves after long periods of time facing situations around abandonment, neglect
and betrayal. I am not telling you this is your case, but it's wise to be aware of all the limitations and needs for improvement your relationship presents, for both of you to work on it.
Thus I believe that it'd be necessary for you to start a time frame to start truly integrating you in each others' lives. If you are truly serious with each other, I do not see any reason for that not to happen like after a year of dating. If you address this issue and he keeps being avoidant or reluctant, then that would be another red flag, or if you agree to do it at that time but he gives more excuses not to do it, then you would need to reassess how well this relationship meets your core needs and expectations.
Does it make sense?