I hope this message finds you well, present circumstances excluded. I am a licensed attorney with over a decade of practice experience. It is a pleasure to assist you today.
If the person that asked you to leave is a person that is a representative of the Senior Center, or has the legal right to speak on behalf of the senior center, then you must honor their request. The legal basis for this is that they represent the property rights of the center, which trump your rights to be there. If you are the relative of someone in the center, then you arguably have a right to be there as well though. As such, they would have to then have a reason for you to not be present that would out weigh your reason to be there.
That said, if you show back up, they can call authorities and have you removed for trespassing.
For that reason, if this center is in a municipality and you have a relative at the center, you need to go ahead and meet with the local authorities (police) about the issue and inform them of what is going on. This way, if you show back up, they will have prior knowledge of you and the issue - which will or should give you the benefit of the doubt in their eyes.
Let me know if you have any other questions or comments.
Please rate my answer positively as well (three or more stars) so I can receive credit for my response.
Best wishes going forward!