It sounds like he is sending you some serious mixed messages. If is very unfair for him to let you be so torn instead of being clear about his feelings and intentions. It is also very troubling that the children don't get along.
You sound like you can approach him without being confrontational (confrontational is never good). You should let him know how distressed you are and the impact of his lack of communication. There's no harm in that. With the right approach, you may be able to find compromises and ground rules that would make you both happy. Make sure you lay your needs out to be resolved. You may be surprised that his stress and obligations may be the reason for his distance and not you at all.
If you do challenge him and he says it's over then he would have said that anyway and everything else is just a smokescreen.
If you really can work this out, you may eventually be able to gradually expose the children to each other and find a way to better mix them which could benefit your situation as well
Be courageous and go for it
Thank you for your help here. I will go and talk to him soon.
If we manage to sort things out and go forward with our relationship, I wonder if you have any suggestions on how the children could get along. The problem seems to be that his son feels threatened and jealous of my three boys living next door to his father, despite reassurance. I'm told his mother has not helped the situation by making comments like "Daddy has a new family now". Naturally this only makes him even more insecure!. Do you think this is something only time will resolve? Any further suggestions would be gratefully received! Thank you.