Are these raised or in the skin like discoloration?
Do they come and go?
Any other symptoms such as fever, etc?
This sounds most like a type of very mild eczema called dyshidrotic eczema or a form of contact dermititis (basically an allergy or skin sensitivity). Because the rash appears to be spreading, albeit slowly, I would really recommend that you have this examined. This could have occured while you were at the gym but this could also have been from a multitude of other exposures... most noteably, if you have never been evaluated for the possibility of a latex sensitivity/allergy, this sounds symptomatic of this. The eczema sounds like your symptoms as well though, and without physical examination and a swab of the bumps, it really isn't possible to defintively diagnose.
If at any time the rash is clearly spreading, you have any swelling anywhere on your body, if you develop hives, changes in breathing or changes in how alert you feel then you need to report to the emergency room because you may be having a severe allergic reaction.
It may also be a different presentation of scabies, which you could have technically gotten from the gym. This usually presents between the fingers so I wouldn't suspect this is most likely but if the other possibilities are ruled out then you may want to explore this.
Your question specifically asks what you can use over the counter to help with this. Calamine lotions will help with the itch but I hesitate to recommend this to you without knowing the cause... a contact dermatitis will likely get better as would a case of eczema but if it turns out to be a bacterial infection then this could exacerbate the problem. Be careful not to apply anything to broken skin or places where it is draining fluid.
I hope this is helpful in answering your question and that you seek the physical exam that you will need for a diagnosis between these possibilities. Please remember to press accept and feel free to contact me if you have anything to add.