Depending on the state you live in, the law protects people who have continuously, openly and "notoriously" (and often accidentally) made use of land from being evicted decades later. Unfortunately, it is not an easily adjudicated matter -- which is why the local recorder said it would need to be taken to court. There may be documents that previous owners signed that allowed the fenceline without giving away ownership
rights -- or a previous owner may have taken legal action to get the 4 inches back. If so, there would be no rights for you to continue holding the land.
However, if no such documents exist, a legal doctrine known as "adverse possession
" may come into play. Again, its not easy to determine if it applies without a lawyer who is familiar with your state and local laws looking at things closely.
It may be that the neighbor will end up taking it to court. In which case, I do suggest that you engage a lawyer. If you are unable to pay for a lawyer yourself, you can often find legal assistance for free or reduced fees. The American Bar Association has an interactive website at which you point at your state, click, and then a list of free and reduced fee legal services pops up for your area:
I wish you all the best in your efforts to keep your property the way its been for four decades. Sincerely, XXXXX XXXXX