I am glad you wrote.
"Holding Therapy" is not sanctioned by the American Psychological Association, American Psychiatric Association, American Counselors Association or any other governing body that deals with therapy and counseling.
That should indicate to you that it is not right, and he should not have been doing this with you. It is, in fact, against common practice and he can have his license and practice taken from him for doing this to you - for the exact reasons you are experiencing now.
I suggest you do two things - find another therapist - one who is not going to abuse you in this way (maybe a female, instead) and file charges against the first guy.
I suspect he knows that what he's been doing is wrong and is just trying to get out of that - he may have been caught or warned - but he needs to be stopped.