I am a psychologist and would be happy to answer your question. Please answer these questions first.
Are you married?
Is this your idea or you partner's idea?
What makes you think it will cause psychological damage?
What do you hope to get out of it?
First, I am pretty sure that 20% of couples are not engaging in swinging. However, I admit that I do not have empirical evidence to support my saying this. I am going to give you an answer based on my clinical experience and watching couples who involve third parties in their relationship sexually. Generally, swinging will not cause "psychological damage." Rather, swinging is often times the result of "psycho/sexual trauma" or "sex addiction." Most individuals who pursue swinging are probably dealing with a host of psychological and interpersonal difficulties. Contrary to popular belief, sexuality involves deep intimacy and sharing between partners. By inviting a third party into your relationship, or by switching partners, you are disrupting the intimate connection that sustains relationships. However, if you are not concerned about disrupting your bond with your partner, I would suggest you give it a try. Your "very secure" relationship will become very insecure after opening up this door. Men tend to become entrenched in sexual addiction after engaging in this type of behavior. Biologically speaking, this is exactly what your man would like to do. If you allow him to do it, he may no longer see a need for a relationship with you. I'm not saying that it will definitely end your relationship, but I am saying that it will change things in a dramatic way. I'm also sure that many other "experts" would disagree with my opinion on this matter. I suggest you seek out further advice before venturing down this path. In my professional opinion, I would not advise anyone to do this...regardless of their perception of their relationship.
If you found this answer helpful, please click ACCEPT so that I get paid for my work. Bonuses are greatly appreciated if you found my answer worthy. Please request me for future questions by typing "question for JR" in the subject line.