If you go to the appointment you'd have the opportunity to speak to someone objective who does not necessarily agree with your husband.
The psychiatrist can also rule this out if your husband is saying that you're paranoid or delusional and that way you'd have some more information to go by when you speak to your husband. You can have the opportunity to be heard out by someone else and you can share what is going on that you know that makes you suspect infidelity. The psychiatrist may also speak to your husband (if you give permission that he goes in with you) and you can hear what he is telling him and what feedback does he get out of it.
If there is a pre-existent condition that can trigger these suspicions then you'd know if that is the reason for your husband wanting you to see someone. If there is no such condition, at least you'd have a professional hear your side of the story and see what they've got to say.