He may know that you have feelings for him and yet he also remembers that despite those feelings he had been betrayed in the past. It is a trust issue on his part now and it is possible that he is trying to get reassurance from you 1) that it wont happen again and 2) that you truly love him as you say you do.
Men can be as sensitive about infidelity as much as women. It wounds their manhood.
If you want to make the relationship work- go for it. If not, let him know. The sooner things are dealt with either way the better.
but what can I do to make our relationship better?