You're in a very difficult situation. You could get a restraining order if he keeps harassing you, but you work with him. You could report him at work, but then it will come out. Even if you were able to do either of those things, he could still figure out some way to let your husband know. There's not much you can do legally, because technically you did have an affair with him, so even a sexual harassment claim would be difficult to explain. You might just want to calm him by letting him know that you are friends, but you're dealing with a lot right now, and if he won't let it go, you might consider just telling your husband thereby taking away all of this man's ammunition. (You might also consider being very upset, and telling him that you already told your husband, and see if he goes for it...tell him your husband is furious and there was a huge argument)
I'm sorry I don't have a better answer for you. I will keep you in my thoughts and I hope it works out for the best.