Ask a Lawyer and Get Answers to Your Legal Questions
I am sorry to hear this and I know how traumatic this can be for the child. Since it was never formally documented it would be hard to enforce in court. You could certainly try and see what the judge does, but it is not very likely. What you can do is file for a modified visitation schedule and explain to the judge the arrangement you had for 2 years and then let him/her ask the mother why she is no longer willing to allow it.
It is hard to predict but based on how the child is responding, it may make sense for the judge to consider it strongly. I think the fact that you pay child support on time and regularly helps your cause.