I'm sorry to hear your family is going through a divorce; it's always very difficult for the parents as well as the children. You'll have to do your best to explain to your son that you and his mother no longer living together, has nothing to do with your love for him and tell him that you miss him on the days you can't be together. Remind him how much you love him and tell him that the next time you see him, you'll do a special activity together, just the two of you.
He's really quite young, and you can make a difference in how he perceives your divorce and how it will affect him in later years, by handling the situation correctly, now. As long as you reassure him that you love him very much and nothing that ever happens will change that, and you miss him and look forward to seeing him on every day you're scheduled to see each other, plus, tell him you're not coming back to THAT home, but you have a new home of your own, where he will be able to spend some time (if in your agreement) in the future, if he's not already sleeping over, now.
Some articles and books re: dealing with divorce with young children, might help you feel less stressed.
As long as you reassure him that you and his mother both love him very much and you'll always be his parents, he should start accepting that, and it should become a little easier as he gets older.
I wish you much good luck.