Thanks so much for your very helpful reply.I am so sorry you are going through this.
From your description I can tell this boy is a nervous, high strung scaredy cat. The females boss him around and probably bully him because they can sense his weak energy.
He has misdirected aggression toward you and your brother because he wants to take the attitude "Ill get YOU before you get me!" This is fear based. I do not think it can be handled with OTC remedies like feliway and rescue remedy.
He needs to be on prescription medication like prozac or xanax.
He most likely fears your brother and leaves him alone because of the punching but I would never advise anyone to hit a cat.
Tell your mom you want to consult a mobile vet to come to the house and get the cat started on one of the above tranquilizers. He cannot be allowed to injure you.
IF you can and want to put in the time to train him let me know and I'll tell you how.
Please let me know what happens. I will always be here for you.
Please be so kind as to rate my answer with positive feedback; that is the only way I receive credit for my assistance. Thank you very much!
Providing a positive rating will not end our conversation, should you need me for any follow-up. Simply click 'Reply' for clarification or additional information, if needed, at no additional charge.