I would avoid contact period especially at work because it could be seen the wrong way and someone might charge either of you with sexual harassment on the job and that is what he is afraid of, you don't want that to happen. It's really important that you leave relationships outside of work. I wouldn't have physical contact with him at all while you are at work. It's best to tell him that the way that he said hurt your feelings and that you understand why but the way it was said hurt your feelings. I would keep your relationship at work purely professional and if you want to get together after work that is another story. Don't avoid him because that will put a strain on not only your personal relationship but your professional relationship even though you are not dating it still hurt your feelings because he already knows what type of person you are and that you like to show affection but others in the office or on the job may not see it that innocently and may take offense to it even though it isn't happening to them some people will try to get higher up on the latter at the cost to anyone and that is why he wants you to show a more professional attitude at work.