If the judge set no visitation and he wants no boundaries, then it is up to you, him and the kids to set these up so that everyone is happy. You are going to have to talk and negotiate what works the best for all involved. There are going to be compromises and not everyone is going to be happy ALL the time, so you all need to realize that to move forward. As far as his money situation, I cannot give you specific legal advice, but you may want to consider looking into getting court issued child support. His living situation is not really your responsibility, but your kids are. You want to make sure they are always the priority and they are taken care of before anyone.
I wasn't clear when you said the kids were inviting him over, but at the same time were afraid when he was around. I'm not sure what their attitude is towards the time they want to spend with him. You need to talk to them about this and get their input. It's very important to have the kids be involved in this process. They will appreciate it for the rest of their lives.