What is your age?
Who are these people telling you this?
The more info you give me the better I can help
The botXXXXX XXXXXne is how you feel about yourself. Nothing else really matters except how you feel about yourself. Your mom is always going to worry about you, getting her to stop worrying about you is like trying to stop the sun from coming up. Unfortunately sometimes parents can become overzealous in their wish for us to be happy, healthy and able to give them grandkids. Ask your mom to stop, tell her that she raised a good daughter and that your life is on the path that it's supposed be on.
As for your co workers, its a thin line between wondering who is or isn't your friend, and whose motivated by jealousy. If you don't feel the need to socialize then you don't have to socialize. If you think that there's something wrong with you or that you can't bring yourself to socialize even when you want to then you may want to consider therapy to see if you can work through it.
I find that people are generally suspicious of anti social people, they will think that you think that you're better than them, that you're stuck up, that there's something wrong with you, that something happened to you, etc. As for your co workers, you don't have to go with them and their families, you don't owe them any explanation. If you need to give them an excuse, you can tell them anything (I was an only child. Its hard for me to make friends. I'm just not very social, etc)
The most important thing to recognize is that you don't know what people are thinking, so don't let your imagination run away with you. If you are different, act different, then people will notice and they may react in some way, thats how people are. Don't assume that people don't trust you or that they think you are inadequate, and if they do, the heck with them, because you know who you are, what you're capable, how you feel and what's best for you.....no one else. I welcome your thoughts, let me know if you want to talk more.
I apologize for using the term so lightly. I did not intend it to mean that you suffered from anti social disorder, only to say that you were not a social person....non social may have been a more appropriate term. As I said earlier, people tend to be suspicious of someone who is too quiet, they think this person may be 'stuck up', or think they are better than them, or that they must be hatching some evil plan. The fact is, not everyone is comfortable with all of the small talk and chatter that people seem to get into so often. You either have to decide to talk a little more, or become a little more thick skinned about the whole think. When you're quiet and not really talking to people to get their feelings about you, its hard to know exactly what they're thinking, so it doesn't do you any good to try and figure out what they're thinking, its only going to make you feel worse, and probably get even more quiet.
I welcome your thoughts, let me know if you want to talk more