I had to switch to Q & A because the computer system would not display our chat to me unless I switched it. But that doesn't change anything. We'll discuss for as long as you need, even beyond a week.
Yes it's a big struggle. I'll say out front that you can't make any choices without hurting someone; that's just the way love is. Also, love is blind, but not entirely without wisdom. By this I mean that 1. the intensity of your love feelings is partially (but not entirely) due to its comparison with your grinding unfulfillment with your husband, and even more so because the relief&release from holding in your disgust & frustration & anger at your husband during romantic and sexual moments with your lover sends the normal euphoria of love moments completely thru the roof. 2. and intensity of love is NO guarantee that that person is Right for you. The most passionate relationship EVER could be SO RIGHT for you in the moment and wrong in the long run of an actual relationship. It also doesn't guarantee that that person is Wrong for you. But meeting and developing your love thru mutual secretive affairs is a good guarantee that the intensity of your love will be jacked up so high that you won't bother to think much about how compatible you would be for each other in the long term.
So I urge you to think about his career, his goals and yours, and issues around children and childrearing, as well as financial issues before diving in.
As to the question "Is it Right?" I hear your unhappiness, and your happiness, and I don't view morality as an absolute in all cases. I weigh the outcomes, for everyone involved. Is your husband a good father to your daughter? How is your coparenting relationship with your husband--ie good for your daughter? How is your lover as a father to his child? Has he divorced his wife? At 6, your daughter could grow up with either 2 good fathers or 1 good and 1 not-so-good, and either way she'd have permanent effects from the divorce, but 2 good & available fathers would be the best outcome, possibly better than you staying in an unsatisfying marriage. Every child grows up with some wounds from idiosyncratic wounded & unconscious parents. If the 1 not-so-good father is her separated biological dad, that's better than if it's her stepdad. Also, if you and your husband don't choose to spend quality time together, then your daughter may be serving as chief companion to Both you and your husband, or at least to You. If you marry a man who you'll really want to spend lots of quality time with, then your daughter will lose her job as mother's indispensable comrade--and that job is not good for her having a carefree childhood. Also, if she sees her dad for limited times of visitation, she will probably have more quality time with him than she does now, unless he moves far away and/or avoids her in order to start his family life over.
There are many more aspects to consider, including your moral values, your relationship with husband's family (how's your lover's relationship with his exwife's family? There's the hurt to your lover and to yourself if you choose not to continue with him, and the hurt to all 3 adults if your husband discovers and confronts the two of you. That's already happened once before, but you can't be sure that your husband's reaction won't be different than yours was.
And there's the ideal outcome for everybody: your husband IS still having an affair with a very durable foundation. So he'd be released to marry the woman he wants, and your daughter could end up with 2 pairs of loving parents, which is better than one. That's rare, but IF your husband welcomes the divorce because he already has someone, then the chances of an amicable divorce and excellent post divorce friendship-collaboration are very good.
So I wonder if there's a way to send up a trial balloon on that issue? Here's a thought (but don't just jump on it without careful consideration). Ask him this: "What if things had turned out differently back when we had our confrontation about your affair during my pregnancy? What if I had divorced you then, or if I'd waited until I was more on my feet after the birth and then despite all the good care you'd given me I had divorced you? Where do you think we'd be now, 5-6 years later, being coparents but not married to each other? Have you ever thought about that?"
If his response show in some way that he's deeply wounded about how close he came to losing you thru that affair, how stupid he was to even risk it, and how he can't imagine ever wanting a different woman than you (since his making amends caretaking may have nourished a more unselfish love than he'd ever known before), . . . . then he might be deeply blind-sided, hurt and furious about considering a divorce now. But if instead he's able to talk about alternative lives he might have lived that sound like they'd be acceptable to what you have together, except perhaps for what a divorce could do to your daughter--then you know that it's something you CAN talk about. And then you could tell him "You know I've been unsatisfied in some ways with our marriage for quite a while, and you don't seem to be able to respond to my requests for change - - - - - - So you could have THAT discussion again, and if he saw that you would seriously consider divorcing because of these differences, he'd either propose new ways to be more certain that the changes you want may be achieved, or he'd be shocked at first, but NOT desperate to make those changes, if in fact he does have his other woman for bailing out of your marriage--unless his secret woman is also married, so more worms would be crawling out of the woodwork.
Have you ever heard how big areas like the medieval Europe, and the Middle East, India and China used to arrange marriages based on astrological charts? I have studied that a little, and I've seen that to some extent it works. If you were to accept that as a possibility, then you could arrange to get the birth charts (birth time & place as accurate as possible--within 15-30 minutes) of both men and yourself and get them compared, as well as composite charts for the relationship. The composite charts will show strengths and challenges in each relationship, as well as what aspects are compelling and what aspects (usually much more of the relationship) are not compelling, so they're up to you to create. If these 2 composite charts should turn out way more compelling and durable/loving with your lover than with your husband, besides showing you many features of both relationships that would be amazingly accurate with no prior knowledge by the astrologer, then you'd have support for divorcing. If the lover-chart should show up a compelling&fatal flaw and the husband-chart show durable love, you'd have support for just working on the marriage, by talking up divorce possibility and demanding couple/sex therapy. If both relationship charts are roughly equally good&difficult, or not very compelling one way or the other, then astrology will not contribute a vote in support of one choice or the other. That might signal though, that the grass isn't actually greener on the other side of the fence because of better nutrients in the soil, though the differences in your histories of grazing on each side could lead you to be better at cultivating the garden next door than what has always been your own.
Well, I've given you a lot more to chew on. OH yeah. You could also call up a psychic and ask him/her if your husband knows what you're up to and if he's up to something similar. But you might want to disguise what you're fishing for, so the psychic doesn't just guess immediately that you're sniffing for affairs and then provide you with one. I don't think most psychics can predict very far into the future; and I don't know about their ethical concerns about pinpointing extramarital affairs. But I do believe in openness towards any source of knowledge IF you can be unafraid of what you'll find and not blown around like a feather by knowledge you can't cope with--such as taking away your basic security (like your religious beliefs if they are foundational for you. Getting a divorce would threaten your daughter's foundation, but if you were very cooperative about it, and got some counseling guidance too, and her post-divorce family environment was stable and collaborative with her father & his environment, she might be better off in adulthood than if you stay in a dry marriage with low intimacy, low love and smoldering resentment between you.