I need some advice please. My ex husband and I had been

Hi there. I need some advice please.
My ex husband and I had been married for 10 years, together for 14 when he left me for a woman he had been having an affair with, when I was 5 months pregnant with our daughter. What followed was very messy, and a very big legal battle.
In short, my ex husband decided he wanted our daughter Mon-Friday, and be allowed to come to my home on the weekends to bath her and put her to bed etc.
He would insist on coming to my home to see our daughter. He would largely turn up when he felt like it (I’d get a text message saying be home in 10 I’ve finished work early), once here he’d lie on what used to be our bed, go through my pantry, even try and use my computer!
I was in a mess. He was manipulating and bullying me. The final straw came when he said he would take our daughter away on holiday wherever and whenever he wanted and I could do nothing about it. On that occasion he also refused to leave the house or give her back to me. He became highly aggressive whilst holding her. That was it for me.
I stopped him coming to my home. He hated this. He had to always be in control of me. He is a very high powered doctor. During our marriage he used to hit me, punch me, pull hair etc, and was extremely emotionally abusive. He told me if I ever left I would never find anyone else because I was a “fat loser”.
Time went on. He dragged me through court, lawyer for child etc, drained me of so much money I had to borrow off my parents. I found out later it was all to punish me (he was on 300K at that time the money wasn’t an issue to him it was a power thing and he didn’t like it I stood up for myself and stopped him coming to the house) as he was planning to go to live in Australia as he had had a job lined up there for months! How could he see her during the week whilst living in another country. It was appalling. He had done it to drain me of money and stress me out. He had no intention of any of what he was asking for.
Since then, I have remarried and we have had another child. Our daughter refers to my husband as “Daddy”. Reason being, he is her absolute father figure. My ex husband sees our daughter 2 or 3 times a year, by his own choice.
We have a court order in place that states he can see her for 4 hours on a Sunday. He agreed to this and never bothered to change it. He was fine about this, (I would always accommodate other days and give him extra time however because I couldn’t handle arguing with him and his bullying).
However he can’t stand the fact our daughter loves her step father so much, that I have remarried and we have a good life. He knows our daughter is my world and he constantly tries to make life hard for us, harassing us all the time, sometimes he would email 10 times until 2 in the morning.
His argument now, is he wants to go from 5-6 hours on one day, to taking our daughter overnight, and ultimately then taking her on holidays etc.
She is only 5. She barely knows him, having seen him only 3 times a year. Every time she has to go with him, she is nervous, and doesn’t want to go, because of how he is with her.
He is now saying if I don’t do what he says he will take me to court and win. He says I have no chance and he will drag our daughter through family court, is that what I want?
He wants to go from 6 hours on one day, to 7 hours Saturday and Sunday and then straight into overnights.
My concern is the stress on our daughter. She comes home down, she never enjoys the visits. He is a narcisstic personality, cold. She must pick up on this.
We feel he is absolutely not in her best interests, however we don’t feel it right for our daughter to deny visitation despite all our concerns. We feel he needs to do things on our terms, as we raise our daughter and know her, and we have always been reasonable and tried to work with him. We just can’t handle the constant arguing and harassment from him. We really try to be nice always but to him, whether we are nice or absolutely horrible (which we aren’t) it would make no difference. He is still rude, arrogant, harassing and bullying.
We are so worried about what he is doing and how it will ultimately affect our daughter. We don’t want her to be pressured, stressed or upset.
He wants only what is best for him. Not what is best for her.
We absolutey believe overnights with him is so premature, she is no where near ready. We asked him to come over more and try and make visits more regular. He refuses.
All I get from him is give me what I want or I will drag you through court. I always try and compromise but there comes a point where I am not doing the right thing by my daughter if I do what he wants only because I am scared of him and going to court, when I know deep down it is not right for her and not in her best interests.
Am I in a strong position? Would he get overnights? He makes me nervous and bullies me relentlessly. Any advice would be so appreciated, thank you very much