The fact that your husband cheated on you not once but twice it's hard to get your trust back especially if he isn't really doing anything to prove to you that he can be trusted again, one thing that dooms marriages after an affair, or keeps a marriage from surviving after cheating would be that the couple were not able to rebuild the trust, even if you decided to forgive and have committed to moving on, the inability to really trust that the cheating will never happen again may be keeping you from trusting him. You begin to wonder what is wrong with you that your husband wanted someone else. Your self esteem takes a nose dive. So you may have put up a wall around yourself to ensure that you are never hurt this way, but here's the problem with that. The wall doesn't allow your husband back in and he has to be allowed back in if you are going to rebuild the intimacy and closeness necessary to rebuild a better marriage and regain that trust that you have lost in him.
You are going to have to let your husband know how badly the affair has devastated you, this must be done because if you don't release these feelings, they will eat you alive and they will always crop up again when you think you have beaten them. You may not have given yourself time to heal from the affair which means asking the questions you wanted to know about the affair the how, when and where if that is what you need to know then ask him but you have decided to forgive him which means you have to let go of the past and tell him that you checked the phone records and want to know who this other women is, I'm a firm believer that "Once a cheater not always a cheater" but if your husband isn't trying to have self control so that it doesn't happen again or if he doesn't show remorse it may happen again so you need to confront him about the fact that you found this number and you called and it was yet another female he will probably try to lie out of it but the proof is there.
Marriage counseling may be your last resort now because things aren't getting better with the two of you doing it on your own. You have to at least try the counseling not only for the sake of the marriage but for your children also. If the marriage doesn't last then you know you tried everything to make it work. Also a marriage counselor can help your husband figure out why you he cheated in the first place and steps you both can take to get the trust back in your marriage.