The children will need contact with their father. But, he would need to call you and make arrangements because now this is your home and he has to respect that.
If you find his behavior inconvenient, talk to him about some flexibility on his end. It would also depend on whether or not you share custody. If you do, he can take the kids to visit him once in a while or see if he can take them out to spend fun time together.
It would all depend on what the two of you as adults decide as long as it does not interfere with your legal obligations.