Yes, you do. But you must give him notice and then, if he still refuses to leave, go to court and file an "unlawful detainer" (eviction) case against him. (You must basically treat him as a tenant).
Terminating a tenancy-
1) Notice: The first step in any termination is giving notice, the landlord can simply give notice that they no longer want the tenant to live there (this is usually 30 days, or 60 days, and it can be done for no reason whatsoever, there is no fault, and while the tenant must relocate, they are not being "evicted" and there is no blemish on their rental history), they can give a "notice to pay or quit" (usually 3 day or 5 day depending on the state, and the tenant has this amount of time to pay rent that they missed or move out), "notice to "cure or quit" (the tenant has breached the lease - broken something, noisy, etc. and must stop it or fix it within the notice period, again 3 days, 5 days, or 10 days), or a "notice to quit" (this is a 3 day or 5 day notice that says the tenant has messed up so badly they can do nothing but move out within the notice period - there is no chance to "cure" - this often happens when there is illegal activity on the property).
2) Unlawful detainer/forcible entry and detainer (this is the legal proceeding where the landlord goes to court and sues the tenant to get possession - the tenant has an opportunity to appear and defend the action, common defenses include improper notice, breach of the lease (such as failure to maintain the property - "inhabitable conditions"). If the tenant answers the complaint, the parties can take "discovery" from one another and get additional information before a court trial before a judge.
3) A judgment of possession/writ of eviction - if the landlord wins the trial, they get a judgment of possession and the court will issue a "writ of eviction."
4) Forcible eviction - this happens when the Sheriff or Constable serves the writ of eviction - some jurisdictions give a courtesy notice the day or two before the eviction, others do not, but the end result is the sheriff overseeing the landlord's movers removing all of the tenant's possessions from the property and placing them on the curb, and the tenants are forcibly removed from the property. At that point, the landlord can change the locks and the tenant can no longer return (They have been "evicted").