That is unfortunate. First off, unless he had either told you that the initial fee would apply for restocking, either verbally or in writing, then he cannot force you to pay that amount in a restocking fee. You cannot charge for fees this excessive without first obtaining the consent of the entity to which you are charging. Since that did not occur, you have the right to demand your money back in total.
Moreover, the fact that you have attempted to deal with him in good faith and the fact that he has not reciprocated towards you bolsters your position.
My advice would be to take him to small claims court for the amount of money that he owes back to you and to have the court to order him to leave you alone as well. Before filing the small claim against him, you need to send him a certified letter with a return receipt in which you demand to be paid back in total within 5 business days of the postmark on the letter. This will serve as legal notice and will help you if you have to go to court.
If he does not agree to pay, then you need to take him to small claims. Remember in court to explain that you were never told you would have to pay a restocking fee and that even if you were, the fee charged was excessive.
You do not need an attorney to assist you with a small claims case. You can do it yourself and based on the facts here, I think you will win with relative ease.
Let me know if you have any other questions.
Please rate my answer positively (4 or 5 stars) so that I can receive credit for my work.
Best wishes going forward!