Hello! It's nice to hear from you.
It sounds like the people who feel the need to apologize to you perceive you as someone who either does not accept such behavior for some reason, possibly judgmental or they see you as someone who is more viturous in character.
It is very insightful that you caught how others communicate their perception of you. How we see ourselves is based on a lot of factors, including how others perceive us. This can be good or bad. If you change your behavior based on negative feedback, you can improve your social interactions and self image. But if you change your behavior to fit in socially based on others' perceptions that you do not conform to their wants and needs, and you give up positive aspects of yourself to do it, then it is a negative change.
How others view you can give you information about yourself. It shows you how your behavior affects others. The people who perceive you as virturous don't know why you act the way you do. So they make assumptions about why you act as you do. These assumptions will develop from their own experiences. For example, if you are assertive in your needs it may be perceived in different ways by different people. One person may see it as a positive trait, another may see it as a negative trait because of their background.
You can change other's perception of you by changing your behavior. For example, you can smile or laugh next time a person curses in front of you. It gives a forgiving or accepting message to the person, who then realizes that you either don't care about the cursing or that you agree with the cursing. Or you can continue as you have been. It's in your control how you want to be perceived in this case.