There are a couple issues here: first of all, any legal action that you would have against him would be based upon "conversion" (the civil side of theft) or his failure to repay a loan. Conversion has a statute of limitations of three years in California, meaning that you could really only sue on the money that he has taken in the past three years. Anything taken outside of those three years would be unreachable because of the statute of limitations. And there would be the problem of "consent", in that conversion requires a taking of property without your consent (at the time of the taking, regardless of your later feelings on the matter).
Failure to repay a loan would be a little better of a cause of action, because the statute of limitations start to run when he has to pay and does not. If you had an agreement that you would lend him this money, and he would pay you back at some other time, you could demand payment, and the statute of limitations would start running from your demand. For an oral contract, that would be 2 years from that moment of demand. But to win, you'd still have to prove that this was a loan, was always meant to be a loan (especially at the time that he took the money) and never meant to be gratuitous. He could always use the defense that it was gratuitous (a gift) and therefore he has no obligation to repay it. That's part of the problem and a weakness in your case.
you can prove your case, that there was a loan, and that he has refused to pay it back, then you stand a good chance of winning. But if you can't prove those necessary elements, then you'll lose. And from an impartial viewpoint, it seems as though you're only seeking repayment because of his infidelity
and indiscretion, rather than seeking repayment on a valid loan. Frankly that leans pretty strongly to the inference that it was gratuitous, rather than a loan.
If it were me, I would ask a few lawyers what they think of my case, and see if they'd take it on a contingency basis. Their answer as to the contingency aspect would give me a very good idea what they really think about the strength of the case. It also might make sense just suing for what you can in small claims court, without a lawyer, because that generally is more "equitable" (based on fairness rather than the law). But even then, I think there's a good chance that the justice would say that these payments were gifts rather than loans or thefts.
I know this is probably not what you wanted to hear, but it's the law. I hope that clears things up anyway. If you have any other questions, please let me know. I do with you the best of luck in this matter, and hope that you can move on from this person and get on with your life.