Have you seen any professional about this?
This would be your first step.
To discuss this with a psychologist trained in family therapy [couples therapy].
That way, you can be clear on your comfort level.
Your wife can be invited into these sessions to discuss these fantasies if needed.
or, you might become comfortable enough to discuss them with her on your own.
Professional guidance can take the conflict you are experiencing out of the equation.
And that is what I would recommend.