Whatever you decide, you must be certain that this is what you want to do 100%
Even if his behavior is boosting his ego, the behavior is not appropriate or respectful to you. The two of you would need to sit down and address this situation. Whatever had propelled him to seek even emotional comfort through another women needs to be addressed. Some people may consider this a form of "emotional cheating"
Try to find out what is it that he is not getting at home that he is seeking in other relationships (is it the safety of having a female companion, texting/communicating sexy messages but not being responsible for a relationship or sexual intimacy, his bored with his family life, etc)
He would have to be held responsible for his behavior. How would he have felt if you were the one engaged in a relationship w/ another man and doing what he is doing?
If you believe that it is an issue of miscommunication with him, mid life crisis, etc. all of the possible causes need to be identified and dealt with. Marital counseling is one option but something has to be done by both of you.
I'm not happy with your answer, we have a very happy healty sex life and we talk all the time, each thing you said I have already thought of and I dont accept you answer and want to withdraw my deposit