It would seem you need to get an apprehended violence order (AVO).
The police should be willing to assist if you have actual evidence of him trying to scare your children (film it if you can, or try to have a witness around when when he seems likely to behave this way).
If the police won't help you, engage a lawyer to make an application for you to the court for such an order.
If the court grants your order it can order your neighbour not to do certain things, such as act in a way that scares your children, and if he breaches the AVO he will be committing a serious crime and may well face a term of imprisonment.
Alternately, you could sue him in nuisance if he is interfering with your and your children's use of your property. This may be a way to get some modest compensation, but in terms of stopping such contact may not be as effective and you will definitely need your own lawyer to pursue such a claim.
You can read about AVO's here:
I trust the above assists.
PLEASE RATE MY ANSWER.