Thank you for clarifying.
What you are experiencing can be caused by multiple things. Social anxiety and the self esteem issues you've already though about. The behavior can also be conditioned. If you're conversing with someone who does not share the same interests as you or if you perceive that the conversation is not intellectually stimulating, you may feel bored (and you may feel obligated to find another topic of interest to talk about) That in return can make you feel anxious (and it is this anxiety that you're experiencing instead of anxiety as a result of your own characteristics)
Behavior and feelings/perceptions can be due to conditioning and the good news is that they can be changed.
Depending on what you have already tried and what you're open to trying in the future, you can try 1/ altering your expectations of the situation/conversation with another, 2/ not going into a situation with preconceived notions, 3/ allowing yourself to feel whatever it is and using it as a learning experience (learning about yourself , accepting your uniqueness in the situation, accepting that your perception/expectation can differ from that of another and that it is OK/without feeling guilty)
Monitoring your self talk (what you think about) during such situations can also shed some light onto how cognitions/thoughts affect your feelings. Then, you can try to change these as you do have control of your thoughts and subsequent feelings. Change is possible and even when it takes time, it is something that can be achieved. Some tools that can assist you in this are NLP (neurolinguistic programming) autogenic training, or even working with a professional such as a counselor or a hypnotherapist. It all depends on the time and energy you'd like to spend in changing this if it is causing you significant distress even if it is caused by an anxiety disorder. Besides medication, some people try to choose the therapeutic or self help route before that.
Change Your Brain, Change Your Life: The Breakthrough Program for Conquering Anxiety, Depression, Obsessiveness, Anger, and Impulsiveness
Daniel G. Amen Md