There are several possible causes of fluctuating behavior.
He certainly may be having difficulty in coming to terms with the parental separation, and this is causing volatility in behavior. Since he still s feeling love for both parents, he may be felling anxious on Sunday night about the impending leaving of his mother, even though he is returning to you. It certainly is possible that the grandparents are making comments to try to alter his perception of you.
It is possible that this will improve with time. However, it is reasonable to try to talk to him about what is happening over these weekends. It is better to ask in open-ended questions, as he may agree with specific situations that you incorporate into questions without fully understanding what you mean by the question. You also can try to get an idea of what is happening over these weekends by certain games, such as asking him to draw pictures of what happened while with his mother and grandparents.
If this persists or worsens, it may be necessary to seek an assessment by a Psychologist.