Have Mental Health Questions? Ask a Psychiatrist Online
I think the hardest thing here for you is accepting your helplessness. Recommending counseling was a great idea, but she's obviously not ready yet. Another idea is getting her some self-help books. Here are a few titles: The Emotionally Abused Woman : Overcoming Destructive Patterns and Reclaiming Yourself by Beverly Engel ; "Boundaries" or "Where to Draw the Line" by Anne Katherine; Bad Boys, Bad Men: Confronting Antisocial Personality Disorder by Donald W. Black [although just because he is abusive doesn't mean he has antisocial personality...but the concepts in the book should still be helpful].
Keep in mind that even if you would rather confront his abuse when you're around him, this may make it worse for you sister and her kids (he may get more abusive). And if the abuse ever gets physical you should consider calling the police (but again, this may make him more angry & abusive).
Try to empathize with why it's hard for her to leave him, but let her know that you will always be there when she is ready to make that move. Take to other mutual family & friends too to see if they have ideas. Even if they at least use the same approach as you, that will be helpful by sending a consistent message to your sister.
I hope this helps. Let me know if you need further advice. Take care.
Please don't forget to hit the ACCEPT button & provide expert feedback.