Thank you for your question.
It's hard for anyone to answer that without knowing this person the way you likely do, and without knowing more. But defamation claims are rarely successful with people in the private sector. That is because in order to prove defamation of character, a plaintiff has to prove that a defendant:
1) Made false statements about them either in writing or print;
2) To another person or persons; and
3) That they suffered damages.
Even IF your co-worker could get people to lie and substantiate that you supposedly said these things, they likely do not have any damages, or not significant damages of any kind. Persons who are most successful bringing defamation claims are those with the most to lose - celebrities, politicians, etc.. - people in the public eye who can be harmed by false claims
I wouldn't worry about a letter they are flaunting in your face claiming they are going to sue you. You might have more concern if you get something from a lawyer - but I think the chances of that are really, really remote. After all, why would you warn someone you are going to sue them before you actually do it? It would be like telling a person you're going to punch them in their face before you do so, eliminating the surprise.
I think they're trying to scare you and upset you, and there's little chance of any actual lawsuit.