If your son is wanting to spend those last years with his dad. And dad is a good father you need to let him. But like you I would feel I didn't want to let him go from me. Sometimes though kids or young adults need that time to finish growing up.
If he agrees to coming home one week out of the month and you can afford it, by all means do so. If you decide to move, also think about how are you going to feel, uprooting yourself from all you know and are happy about. And check with him maybe he might want to keep ties to Virginia too. Butif you and dad have a good relationship that can make it easier for your son, by all means move. I know personally I wouldn't want my son, who I have raised for years to be suddenly gone accept for 1 week a month.
Its whatever you can handle, one day he grows up and he will have his own life. A wife a family. So make sure whatever your decision its one you can live with too.
If you have more questions on this , just ask, I will be happy to help !