He may need time to work through whatever he is going through right now. Anytime there is an affair in a marriage, the other partner goes through stages as they work through their feelings about the affair. He may be very much in the anger stage now and feel he cannot cope with seeing you because it reminds him of the pain. That is normal. He just needs to work through this stage so he can move on to the next.
You can try to continue working on this issue yourself. Let your husband know you are still trying, even if you have to tell him by email. Keep saying you are sorry. The reason for saying this is to remind him that you made a mistake and that you still want to try. If he will no longer see a therapist with you, go by yourself. Also, try educating yourself on how to repair your marriage and recover from the affair. Here are some resources to help you:
Not "Just Friends": Rebuilding Trust and Recovering Your Sanity After Infidelity by Shirley P. Glass and Jean Coppock Staeheli
Transcending Post-infidelity Stress Disorder (PISD): The Six Stages of Healing by Dennis C. Ortman
How to Help Your Spouse Heal From Your Affair: A Compact Manual for the Unfaithful by Linda J. MacDonald M.S.
You can find these on Amazon.com or your local library may have them for you.
It is hard to keep giving your husband space, but try. Anytime he is willing to talk with you, let him know you intend on continuing to work on the marriage. Ask him if a temporary separation would help instead of considering a divorce right now. See if he is willing to bend a little. Time is needed to see if he may feel better and be able to continue to work on your marriage.
I hope this has helped,