Hello and and thank you for your question. For over 34 years, I have answered questions just like yours. And I look forward to assisting you.
What you are describing is unfortunately very common as the SSA makes many, many errors. Hopefully you will be talking to a supervisor. And, if the error is not corrected, before leaving the office file two forms: Waiver of Overpayment and Request for Reconsideration. This will move your appeal of their decison up to the next level where typically a person with much more knowledge will reveiw your claim. I wish you well.