The botXXXXX XXXXXne is how you feel about yourself. Nothing else really matters except how you feel about yourself. Your mom is always going to worry about you, getting her to stop worrying about you is like trying to stop the sun from coming up. Unfortunately sometimes parents can become overzealous in their wish for us to be happy, healthy and able to give them grandkids. Ask your mom to stop, tell her that she raised a good daughter and that your life is on the path that it's supposed be on.
As for your co workers, its a thin line between wondering who is or isn't your friend, and whose motivated by jealousy. If you don't feel the need to socialize then you don't have to socialize. If you think that there's something wrong with you or that you can't bring yourself to socialize even when you want to then you may want to consider therapy to see if you can work through it.
I find that people are generally suspicious of anti social people, they will think that you think that you're better than them, that you're stuck up, that there's something wrong with you, that something happened to you, etc. As for your co workers, you don't have to go with them and their families, you don't owe them any explanation. If you need to give them an excuse, you can tell them anything (I was an only child. Its hard for me to make friends. I'm just not very social, etc)
The most important thing to recognize is that you don't know what people are thinking, so don't let your imagination run away with you. If you are different, act different, then people will notice and they may react in some way, thats how people are. Don't assume that people don't trust you or that they think you are inadequate, and if they do, the heck with them, because you know who you are, what you're capable, how you feel and what's best for you.....no one else. I welcome your thoughts, let me know if you want to talk more.