Generally, you can only ask to have a custody order modified if there has been a "substantial change in circumstances that affect the welfare of the child." This is a high burden.
It sounds like the judge wasn't too happy with your ex's lawyer during the conference call. The attorney didn't follow the rules for petitioning the court, nor did the attorney, during the conference call, give the judge any reason to change the order.
Unless the "restricted person" has done something dramatic to change his position in the eyes of the court, he will remain a "restricted person." Your ex has the burden of showing that there has been a "substantial change." Has there been a change in this person's lifestyle? a change in his behavior? a change in the persons with whom he associates? a change in his use of drugs and/or alcohol? etc.
You may wish to contact a lawyer who specializes in family law to discuss the specific facts of your case. Many times, an initial consultation is free. You can go over the existing order and any claims that your ex is trying to make. You can learn your options, and then decide whether or not you need to hire an attorney.
It sounds like you are in a good position, but you can never assume anything when it comes to judicial decision making! (as you probably already know)
I hope you find this information useful and that you PRESS THE GREEN ACCEPT BUTTON so that I can receive credit for answering the question. Thank you.
Legal Disclaimer: This is for informational purposes only and does not constitute legal advice.