-- You and your husband need to present a united front here. Set boundaries. This couple says they are "ok with it" in that you and your husband do not intend to partake of their personal lifestyle. Yet the husband of this couple is persisting in asking you uncomfortable questions.
The next time he even starts along those lines, you need to stop him mid-sentence and remind him that he and his wife assured you and your husband that they respect your sexual preferences and decision to remain monogamous; yet here he goes again.
If he cannot honor his word to honor your choices and your right to live the way you want to, then you are going to have to reconsider how honest he is in other areas and whether or not you have anything in common.
Then it's your and your husband's choice to continue or not. From what you've said it may be a good idea to move away from this couple before investing anymore time into the 'friendship'.