If the sheriff cannot determine the property boundaries, then the sheriff will back away and tell you both that this is a "civil matter," and you'll have to work out your differences in court. Law enforcement cannot enforce a criminal trespass claim without clear boundaries (which means surveyor marks). A sheriff's deputy is not a surveyor, and the sherrif won't hire one to resolve the dispute.
The no trespassing signs will not matter, because you can claim that all of the tress are on your side of the boundary line, and there will be no means of determining differently.
In the worst case, the sherrif would tell you that you can''t remove any of the trees, because no boundary is apparent. If that happens, then you will have to hire a surveyor to plainly mark the boundaries. Then you can remove the trees as I previously described, and/or sue the neighbor for damages and/or an injunction for the nuisance.
You can sue in small claims
, without a lawyer, only for money, and only up to $5,000. If you win, that ought to be sufficient leverage to permit you to get the neighbor to remove the trees. If you sue in regular civil court, then you can get an order to remove the trees, but it could cost you a lot of legal expenses to do so. The neighbor would have similar expenses, which hopefully would stimulate a settlement.
The final recourse would be that if the neighbor shows signs of mental illness when the sheriff is present, then the sheriff may just take the neighbor into protective custody for observation -- or contact Adult Protective Services to determine if the neighbor needs to be placed in a nursing home.
Which would solve all your problems.
Those are the possibilities. Hope this helps.