Sometimes this happens if the prescription is too strong, it makes your eyes feel more strained and swollen. When you take them off, the feeling goes away.
Or it could be that the frames are touching too close to your face and causing a reaction.
You should check with your doctor to make sure you are using the appropriate strength.
Hope this helps and thank you.