Yes, if you feel it is best for you to stay away then you should. Especially in the case of abuse, it is important that you feel you have the ability to stay safe and that you have time to heal. Abusers often try to reconcile with those they abuse because it lessens their own anxiety. It also gives them more control over their victims.
Until you feel comfortable and safe, you should not return. You should be able tell when you see his behavior change, and you just don't hear it from him verbally.
Let me know if I can help any further,