Your question was can he sue me, well, sure, he could sue you. I suspect the better question is would he win a lawsuit against you. While I think it's probably a close call, my thought is that you would prevail if he sued.
I would suggest that he has several problems in a lawsuit against you. One problem is the lack of a written contract. While an oral contract can be enforced, reducing it to writing certainly makes enforcement easier. Further, it does not appear that you started the contract in any way or that he relied on you in any way.
Please let me know if anything requires clarification.
I hope you found my answer helpful. If so, please click on the green ACCEPT for my answer. This is necessary for me to be paid for my work and so that I can get credit for assisting you. Even if you are a subscription member, you will need to click ACCEPT. Your question will not close, and you will still have the opportunity to follow-up if needed. Leaving a bonus and positive feedback is not required, but doing so is certainly appreciated! Thank you and good luck!
Also, several customers have asked how they may direct a question to me in particular. If you specifically want me to assist you in your legal matter, just put "FOR JOSEPH" in the subject line and I will gladly pick up the question as soon as I am on-line.