Criminal Law Questions? Ask a Criminal Lawyer.
Technically, they could charge you with conspiracy and/or aiding and abetting, or even contributing to the delinquency of a minor (ie your daughter), but I think that is somewhat unlikely. If they do decide to charge you, however, you should get an attorney as soon as possible to defend your interests.
If he has a stay away order in effect, he is not allowed to be around your daughter and doing so violates the terms of his probation. If he wants to be involved with the baby, there are avenues he can take that are not illegal, such as seeking visitation outside of the presence of your daughter (a court may appoint someone to supervise the visitation). It is up to him to try to take advantage of these legal options, however. Out of an abundance of caution for your own liability, I would not sanction his visits any more.
Hope this helps.