How old is the cat?
Can you give some details of the cat's preference?
Olive sounds adorable. I think it's clear that she is a little bit afraid of the other person. You mentioned the loud voice. Cats are very sensitive to sound, and a loud voice can be frightening. A bigger person can also be scary; men are often more frightening to cats than women.
Olive probably is a little shyer than your others because of her history. Cats also have individual variation, just like people do. Some people are more shy, some cats are the same. Obviously, she is very attached to you becasue you raised her.
What I would recommend is that the other person sit or lie on the floor near Olive. By being close to the ground, it's like making oneself smaller and less scary. Standing up, we humans look m,uch scarier to a little cat on the ground. Speak very quietly and wave a peacock feather or similar toy. Have a treat handy and give it to Olive when she gets close. Don't try to grab her or hold her. It has to be totally on her timetable, but with patience I think she can grow to love your housemate as well as yourself. Continue with daily sesssions like this, and I think you'll see progress.
I hope this was helpful. Please feel free to ask any follow up questions you have. I would be happy to talk further with you.
Thank you for accepting my help, and for that picture of your girl. I was right - she is totally adorable!
Best of luck to everyone in your house in the new year. Little Olive is lucky to have your care.