You're correct- you're not the one making him miserable. Does he even know what does? Could this be related to depression or work dissatisfaction/burnt out?
He is already telling you that you won't forgive him and that is making an assumption and self sabotaging himself from the start. From what you shared, it seems that when this happened in the past, he regretted it, as though he had acted impulsively at that time. Do you believe that he may have some emotional instability such as bipolar disorder traits, and does things w/o thinking about the consequences?
It would be best that the two of you talk about this in person for a couple of days. You may try to find out if he knows what is going on within himself to make him arrive to the conclusion that the two of you should not be together. Is he wanting something different? Is he complacent or just unhappy w/ himself and not wanting you to be exposed to that?
The two of you would want to discuss what in your opinion had changed over time for the last 6 years. Did your mutual goals of the relationship change or your individual goals now clash with each other's goals?
Even though he is stubborn, ask him what does he propose as a way of addressing this issue w/o rushing into decision that would be regretted later on. This is a life changing decision and would affect both of you. If you give him the control of coming up with some doable/reasonable solutions, he may feel more empowered and focus on that versus on how you allegedly won't forgive him. He may have said that to you as a way to deflect the attention from him and place it onto you. But, like you said, his misery is not because of what you're doing or who you are. It is self generated and he'd have to find and address the cause(s) behind it.
You may want to write down some of the things you'd want to cover with him when he is back from his work trips. Let him know ahead of time that you would like the two of you to put some time aside where you won't be distracted and will be able to go over things that need discussing. You can suggest to him that he also writes down some of his concerns.
Remind him that the two of you had invested a lot of energy, time and feelings into this relationship to just let it slip away w/o even trying to manage it. He may have not thought about the long term outcome of this should he leave the marriage. What he is seeking may not be found out of the marriage either.