Is he also your wife's doctor?
From the standpoint of medical ethics, it is a clear violation for a doctor to have an intimate relationship with any person with whom there is a doctor-patient relationship. There is not an equivalent proscription against having such a relationship with a family member of a person with whom there is a doctor-patient relationship. There would be concern if the physician took advantage of his professional position as your doctor, but if the source of the relationship was their respective positions at the hospital, there would not be the same concern.
From the standpoint of general ethics, many would consider that any affair with a married person is not ethical, but that is not the same issue as would be a violation of medical ethics.