Hi. First and most important you are not stupid for having hope and wishing things could turn around. He hasn't given you any indication that things would be different and staying with him you also get all of his problems and issues with his (ex) wife. You need to ask yourself if this is the life you want for yourself and with someone that doesn't seem to be able to give you what you need emotionally and physically. Only you can decide whether to break it off or not, but from the outside looking in, I do not see a wonderful and mutual relationship. It seems quite messy...if you can handle all of it then great, but if you would like something less complicated and more mutual, then your answer seems clear.
How else can I support you?