3 1/2 years is a long time to be together to end it so suddenly so I would not stress that a break means the end of you and him.
The fact is this is a VERY stressful time for him. He is likely dealing with a soon to be ex who is giving him a hard time everyday asking about you. Add the fact that the house will not sale and you are ready to settle down and this is heading towards a melt down for him.
The reality is these things are common when you are with a married man, the fact is he is feeling horrible because he has hurt his wife. I know he cares about you, but you have to accept the fact that he did marry her and vow to be faithful to her and now he has hurt her.
She is likely pushing him about all this as well, and likely hounding him about his morals and character. To be honest the fact is he cheated, and not just a affair but one of emotional pulls as well. Many people would question his character that he would do this to his wife.....who he vowed to stay faithful to. Keep in mind he likely has her breathing down his neck, her family, his family, friends etc all looking at him like he is the bad guy here. And the fact is that is enough to cause serious stress.
The good news is these things will work themselves out in the direction they are meant to go in. But if you are also pushing and pulling at him he is going to resent everyone......maybe even you at the end of the day. You can not force him to do what you want, but you can offer to be understanding and loving by respecting what he wants.
If he needs space, then I would reccomend giving it to him. In the end if he feels you did not force anything then he will not resent you. The fact is he is going though all of this because of his relationship with you.....the last thing you want is for him to rationalize that if you were out of the picture then everything will be OK.
By taking a step back and giving him the space he needs he can make these choices without pressure and by himself so he never feels that he was pressured into something he did not want.
Men can be very funny when it comes to questioning how they ended up in these situations and unfortunately some men do blame the mistress. By giving him the space, he can not blame you as he will be making his own choices.
I don't think a break with distance you as much as you fear, but I do think it will allow him to clear his head and make the choices to do what is best for everyone including you and him.