Hopefully your adopted son had shared his feelings about this potential meeting. Perhaps it would be helpful to approach it from his point of view and see what his expectations may be. As long as each party's expectations are realistic, in the event that something disappointing takes place, everyone will be prepared.
It may be a better idea to have your adopted son meet with her alone initially and then if she and the rest of you is willing to meet her perhaps do it on a different day. That way you will not feel as stressed out.
You've mentioned that they met few years ago, hopefully they've communicated or kept in touch so as to not feel disconnected at this new meeting that is coming up.
I think that your husband has his own reasons for his feelings towards her, of course, objectivity is what usually makes things easier (seeing things from the other person's perspective in this case the biological mother versus focusing on his own feelings)