I am sorry, I do not have good news for you. Since she is her parent thus it is your sisters responsibility to care for her. If she kicks her out she can be charged with abuse and neglect. A parent is required to care for their children regardless of how hard it is.
There are programs she can be enrolled it. Boot camps for teens as well as more stringent private schools that require her to stay on the property. Military schools also seem to help get kids this young back on track. From what you are describing this is not a bad kid.........simply a kid who picked up some habits and is testing her boundary's. .
The steps she has taken thus far as good. She may want to consider handling this in house before attempting to involve the law. The fact is unless her daughter is breaking real laws or is violent with your family the police will not become involved. The courts are full of kids who are looking at jail time for adult crimes the judge will not want to deal with a behavior issue.
Allowing her to get her own way will not teach her any lessons. She may want to consider some strict rules and regulations. Such as she is forbidden to go anywhere until her grades are back at a acceptable range. Some parents have found that stripping the child of everything often helps them to earn the privileges. As a parent she is required to provide a roof a bed and food. A TV, toys, name brand clothing, cell phones video games are not required by law.......I have seen parents strip everything from a room but the clothes and a bed and linen and force the child to earn these things back. A child can only be defiant if you allow her to be. The fact is kids are comfort freakes...they like having things and often if they are forbidden from going anywhere and have nothing to do they often decide that maybe the route they took was wrong.
If she really honestly has tried these things with no progress on her part.(And given it a adequate amount of time.........at least 90 days) then she may want to consider something a little more drastic such as military school. There are some wonderful programs where kids can and do succeed out there. Make sure you do research on the schools and determine what ones are best for your family. Most work on the privilege system. The child must earn the privileges she wants.......until that time they have none.
As far as the courts until she actually breaks some adult laws or is unruly to the point where she feels she can no longer control her. (The courts will require you to show proof that she is doing something really bad). If this is the case she may have a chance of getting her removed from her custody and placed in a detention center. This is something she will want to think long and hard about.......the old saying. A kid went in and a criminal came out is so true in these detention centers. There are hard core criminals in there regardless of age and this child will be around them. A young girl is influenced by those around her.......put her with real criminals and this will be a issue. There are thing that happen in detention centers such as beatings, rapes and much more that you do not want to subject this child to unless you have no other choice.