The argument on the government side is that [don't get mad at me] all your room and board and living expenses were paid by the government so that your social security retirement income was suspended but not medicare for medical. It is similar to someone is in long-term care facility paid by the government on medicaid, all social security income due to him/her is then paid over to the government and the individual is only allowed for monthly allowances for like $30 per month.
Please take a look of this 10 page publication from the SSA on the subject matter.
Please make sure that you receive your social security currently, correctly to the largest benefit possible.
I don't know any case law to your argument. There is a good website for you to research further on case law.
That is Google Scholar. It allows for search on line free all cases and scholarly articles.
Please feel free to follow up.
Fiona Chen, MPA, Ph.D., CPA, ABV, CFF, CITP