"Am i crazy to think this could work?"
Things can work out as long as he's willing to address what is going on. He had given you mixed messages and now you'd want to make sure that he is being honest. It is quite possible that he had realized that he does not have a future with her and her kids and wants to get his life back with you. He would want to tell you how is he going to address what had transpired, own his behavior, come up with ideas about what the two of you should do ex: get counseling, take things slow, have him move back in, etc.
You can decide to trust his words as long as he backs them up with his actions. The two of you would have to talk about what had led to this on his part, what did he expect or want from you and from the marriage in general. What drove him to be unfaithful and to have gone this far. Some of the things may be difficult to hear but he has to be honest with you and with himself. If you believe that he truly wants to work on the marriage and that you can find it in withing yourself to forgive and give him a chance, then it is not crazy to do so.