I thougth this was in relation to the father. As grandparents, if you have not see the child for one year, the 3 weeks is just about the right time for a three year old to recognize you once more as grandparents. But you have been talking to the child once a month ( i recommend every three weeks), which will not be so bad. the child should remember you once he makes the recognition of the voice while watching you speak.
All children go through this and it is not harmful. It is normal. Because of the child's memory configuration and processes, they are not able to remember that the grandparnets are who they are until between the ages of 3 and 5. (some sooner).
So for infants and toddlers this is common and expected.
Grandparents, while imporatant, do not have a right to custody or visitation, legally, except by what is given by the aparents, of under certain circumstances what the courts allow or direct.
Hopefully you would agree with me that the parnets have a bigger need to maintain contact with the child.
the main issue is the father appears to be protesting the child in your custody while he and the mother work things out, based on permanent damge to the child frm not seeing the father for 3 weeks. permanent damage is not proven. But, for the good of the child, the father should be allowed to speak to the child at least once in three weeks, and once a week under these circumstances would also be relevent.
In actuallity, the father can legally take his wife to court and fight this. The mother has no right to transfer custody of the chlid away from the father without a court order.
We can get very detailed bout what can legally be done or what is for the good of the chlid.
Your basic question was related to the father's contention of permanent damage to the child after a three week absence. This does not hold up to scientific scrutiny.
What does hold up is the recognition factor for three year old, such athat weekly phone calls would be in order, or at least a bi-weekly phone call.