As I've indicated, the law is clear as to when you can use force. I agree that the facts aren't always clear as to when force is reasonable, however, which means you can be charged with harassment but whether or not you can be convicted will depend on what the facts make out.
To avoid this problem in the future, there are several things you can do, all of which wouldn't involve force, and would require her to be accountable if she didn't stay out of your house.
You could contact a lawyer, for example and have him write a cease and desist letter putting her on notice that if she continues to come into your house without your permission, you will take legal action against her. Alternatively, you could talk to the police and see if they'd be willing to contact her for you and to let her know that the next time you complain to the police about her unauthorized entry into her home she will be arrested for trespassing. The police are usually willing to do something like that because it makes their job easier.
You could take the matter into family court and get an order which will require her to pick up/drop off the child at someplace other than your home. You could get a criminal or civil restraining order barring entry to your premises.
So there are things you can do to solve the problem. Physical force of any sort, however, is not one of them, unless it is in self-defense.