A mother can never worry enough. That is what mothers do. As far as if there is any substance to your worry, that may or may not be the case. If he enjoys going to the parties that is an indicator that others are not overly concerned about his speech. Kids usually like to have fun. A speech difficulty is not contagious so this may not be the issue for invitation. You had mentioned that he seems reserved. This may be more likely of what is going on. Of course, the more he interacts with others, the more he may like to do it. You won't be able to change a personality pattern if he is more introverted but even introverts enjoy the company of few close friends.
If he has not used the services of a speech therapist, you may want to contact your medical insurance to see if they cover the services and to get referrals.
You can also see if kids can come for a sleep over to your home (if that is possible) and he can invite some of his peers that he is friends with.