Thank your for asking Just Answer,
First, given your ex's track record for visitation and custody then you should not be concerned over a Court deciding to give him joint custody. Courts do generally favor as must visitation or shared custody as possible but his lack of really giving an effort through the years does not lend itself to convincing a court to give him more custody.
Second, actually getting a court to take away parental rights and give them to your current husband is very difficult. Generally a court will have to find no communication or visitation for years as well as failure to pay any child support. Even the minimal visitation that your ex has had over the years is likely enough to prevent him from having his parental rights taken away.
Hope this information helps. Please click Accept so that I receive credit for my response. A bonus is a great way to say "thanks."
DISCLAIMER: Answers from Experts on JustAnswer are not substitutes for the advice of an attorney. JustAnswer is a public forum and questions and responses are not private or confidential or protected by the attorney-client privilege. The Expert above is not your attorney, and the response above is not legal advice. You should not read this response to propose specific action or address specific circumstances, but only to give you a sense of general principles of law that might affect the situation you describe. Application of these general principles to particular circumstances must be done by a lawyer who has spoken with you in confidence, learned all relevant information, and explored various options. Before acting on these general principles, you should hire a lawyer licensed to practice law in the jurisdiction to which your question pertains.
The responses above are from individual Experts, not JustAnswer. The site and services are provided “as is”. To view the verified credential of an Expert, click on the “Verified” symbol in the Expert’s profile. This site is not for emergency questions which should be directed immediately by telephone or in-person to qualified professionals. Please carefully read the Terms of Service (last updated February 8, 2012).