Hi, I'd like to help you with your question.
It sounds like you and your husband have a great relationship and that you are willing to let him express himself as a person. Lots of relationships do very well when both people are so compatible.
However, although it sounds like you might be ok with the cross dressing, your husband has taken it a step further and involved other people. This may also include deception being that you feel he has lied to you about the intensity of the contact he has with others. Deception can undermine the trust in a marriage and since that is part of the foundation of a marriage, it can cause much damage.
Also, the differences in how you both view this contact with other cross dressers is where some of the conflict is as well. He views it as no threat to the marriage and you do. His refusal to see your side of the issue adds to the problem. When two people are married, they have to find a balance between being individuals and making adjustments for the feelings of the other person. Also, marriage is a vow between two people and that vow includes not involving others in the relationship. You are not being put first and that can cause marital problems.
If talking with your husband hasn't gotten you anywhere, I recommend counseling. He needs to understand to put you first and to eliminate contact with these other people. You can talk with your doctor for a referral or if you attend church, your pastor will be able to help. Also, your local community mental health center will know resources for you.
There is a book you may want to try called My Husband Wears My Clothes: Cross dressing from the Perspective of a Wife by Peggy J. Rudd. It may give you some insight into the issue and when behavior begins to cross the line.
I hope this has helped you,