Family Law Questions? Ask a Family Lawyer Online.
Keep obeying the court order. Make a record each time he comes without a licensed driver, so that if he ever takes you to court, you can show the Judge evidence that you were just following the order. You are doing the right thing. It's not your fault that he can't seem to obey the order. If he gets upset and trespasses on your property or makes you afraid for your safety, you can call the police and they will come talk to him and tell him to leave.
He must agree to switch days and you should get that agreement in writing (text or email is fine). You cannot make that choice for him. If he has court ordered visitation, you have to make the child available to him or risk being in contempt of court. Now, if he agrees to take a different day, that's fine, but if he doesn't agree, he gets visitation as court order (provided he follows the order and comes with a licensed driver.)