While you say that your sister is not a danger to herself, the fact that there are feces and urine all over her house suggests that she is unable to care for herself. In some states there are provisions for hospitalization for people who engage in "self neglect"-- this is something that you may want to look into in your state. (I would call the district court in your area to ask about this.)
If your sister really doesn't meet any criteria for involuntary hospitalization, you might try asking your sister to participate in a family therapy with you and your mother. You can frame it as your own struggle with certain issues and ask her for her help. This may or may not work, but if it does get her in the door, any good family therapist will be able to pick up on the issues and concerns that you are having and will help you figure out how to proceed from there.
Good luck to you!