It sounds like your grandparents have their own issues and they are putting them on you. What they are saying is not about you. And what they are saying is very discouraging. If you have been exposed to them your whole life, then it may be why you question yourself now. With that kind of input, you are going to feel that you do not deserve to be happy or even deserve to try for happiness. But you do.
Your grandparents opinion is just that, an opinion. They cannot predict what will happen with your relationship and they certainly know less than you do about it. You are in it, and you know what passed between you and this man. You know how you feel about him too. And that is important. Your feelings are real. And you deserve a chance at this relationship. Will it turn out perfect? No one knows. But to not even try would be much worse. Then you will always wonder what could have happened, and that would cause you pain the rest of your life. At least by trying, you will know. You can live with the results regardless of how it turns out. If you do end up together, then you will be happy. But even if you don't, you will mourn the loss and move on. It won't be easy but you will at least have tried, and that says that beyond all discouragement, you made the effort.
God is there for you. You can pray to him for the best turnout. He knows you and he understands your grandparents. Your grandparents can pray all they want for whatever outcome they wish for, but God is in control. If this relationship is meant to be, then nothing can change that, even someone praying against it.
Don't give up. You can do this. I will be here for you no matter what happens.