Your gut is perfectly right on all accounts. It is poor boundaries of your wife to invite any man to spend the night in your home without your agreement - not just your consent or notification. It is very creepy that she does this, and that he does it too. I've done family counseling for 28 years and have never heard of a family who did this, or wanted to. You're not a jerk for having this move towards normal, it's just normal. You can say, "our kids are getting older, and I'm ready to make new arrangements." The guy can come over in the morning...if you want him too.
I think you and your wife would really benefit from a couple sessions with a family counselor to learn about and decide on appropriate boundaries for the blended family you have. You don't want to end up as the bad guy, and she may take it like this is a personal issue that you have with her ex, instead of just a normal transition through the structural change that happens in divorce and remarriage.
So, back to your questions:
obligated to honor the ex marriage in your home? no
are you the man? yes
do you have a say? You better have, or this is really a much bigger problem
wrong to feel this way? No. normal
let it go? I wouldn't, nor do I recommend that you do.
is it inappropriate? yes
uncomfortable? yes, and it really will be odd for your kids to have this open back door all the time.
You make perfect sense to me. Talk to her calmly about it, and if that doesn't work, go to counseling for a couple sessions and set some ground rules for the marriage.
If you would, please fill out the feedback form after accepting. I appreciate this opportunity to help you out today, and I've done my best to give you a direct, informed response to your question. If you feel my professional answer is worth more than your original bid, a bonus is the way to show it. If I can be of further service to you, just put "for Anna" in the front of your new question, and I'll be the one to answer it.