I am sorry to hear you are worried about Orange. My name is Rebecca. I am a veterinarian and will do my best to help.
This is not unusual behavior for cats to hide if strangers are present; it is the unusual cat that goes up to greet people he doesn't know. My own cats are invisible for a day or so when visitors come but since my cats are indoor cats, they hide in the house. It has nothing to do with being harmed ever; many cats are just cautious by nature about new things, new people, new food, etc. He probably has a hiding place outdoors where he feels safe, and it probably is close by your house.
If you want to keep him in, you will have to close doors and cats doors and if that is not possible, make him as comfortable as possible outside: heated cat house, water replaced if it freezes, food, etc.
You can try to desensitize him to visitors: Put Feliway diffusers in the house; they emit a cat calming pheromone. Provide good hiding places: cats like to be up high to feel safe so having a shelf up off the ground with access by other shelves or steps, in a room he likes to be in such as your bedroom may help. Have a cat friendly person come over and have cat treats he really likes on hand and have all exits closed so when the friendly stranger comes in he can't escape, and keep throwing treats and catnip to him to try to make him associate guests with rewards.
It is difficult to change a cat's basic nature; I once had a cat for 19 years that hid even from me and only came out at night to be petted when we were lying down or sitting.
Let me know what else I can answer or help with.