You may have a sexual addiction. This is identical to drug addiction. It is the inability to stop any behavior despite the negative consequences. This has become a very legitimate problem and therapists have begun to specialize in this problem. If you are interested in saving your relationship you can find someone to talk to. This can help you identify why you seek out inappropriate sex and what purpose it serves. You can also identify why you do this frequently when you are in a relationship. There is usually a payoff to taking a chance losing your relationship. This is more than just seeking outside comfort. Before you look at your history you have to find out why you can't keep your promise to your wife. There may be some truth to the abuse you have suffered but all of this has to be discussed in a professional setting. You have to really explore all of your issues.
If this is helpful press accept