I think you should ignore the comments of those people. As you correctly say...they are just blaming you for the bad conduct of your husband. It is easier to do that than face his failures and their inability to help him. Don't take that negative talk in....IT'S NOT ABOUT YOU. Plus it saps your strength to be worried about what they say...and the fact is you need all the strength you have to deal with your problem. That is...looking at your marriage and your husband and deciding your future. Your kids can't be benefitting from their dad's actions. Perhaps a separation or divorce will help him see the effects of his past and will help you see more clearly what you want to do with your life from here on out for your own good and the good of your children.
I think it is time to remove your children and yourself from this man....get a divorce lawyer and move on with your life. You sound like a strong woman. You have lived through hell. The divorce will be nothing compared with what you have gone through.