Thanks for bringing your question to JustAnswer.
It sounds as if he is reliving the abuse he suffered from his father..by acting out, and then cowering and screaming not to hit him. He may be aggressive toward his mother as he may feel she didn't protect him from the abuse. It's very common for a traumatized child to be angry at the non-abusing parent.
This is more of a psychological problem than a behavioral problem. He needs therapy to work through the feeling he has from the abuse. Until he gets psychological help, he will be unlikely to be able to control his behavior. I would also suggest that the younger sister than he targets get some counseling too....I have treated several adults who were traumatized as children by their siblings.
Please do what you can to convince his mother to get him help. He is carrying around a lot of anger and hurt, and his acting out will likely get worse as he gets older if he doesn't get the help while he's young.