As conservators of these children, because the mother's rights have already been terminated, and because the father's problems evidence that he is not a fit parent, you and your husband can file a petition to terminate the parental rights of the father in the county and state of the residence of the children.
If you can prove that the father has neglected, abused or abandoned the children, you've got grounds to file suit. Also, the fact that he has been incarcerated gives you grounds.
You should contact a local attorney about filing this action. You could also seek legal help and advice from your local child services department.
Lawyer
County Prosecutor