Well, the fact that it comes and goes seems to make a reaction to something less likely.
(An allergic reaction to something would generally cause a more constant numbness, along with swelling of the tongue that would worsen, then get better).
And stress can certainly be a factor, as people with anxiety can often have sensations of numbness, and numbness in their fingertips, or lips or extremties is often a common symptom of people who have anxiety attacks.
But certainly, it would be helpful just to bring up with your doctor, since an examination of the oral cavity and your tongue can help to ensure there isn't any issues with the tongue itself that could be causing intermittent numbness, since if there was an issue with the tongue causing numbness it would be seen on an exam.
Also, the doctor can do bloodwork to ensure that all of the electrolytes and other potential issues with vitamin and hormone levels are normal.