You should review your land records to see what type of easement (rather than just for the purpose of ingress and egress) you have. An easement for ingress or egress generally means just the ability to enter and leave a plot of land. In addition to that, however, the easement itself may be any of the following types:
Necessity - where large parcels of land are subsequently subdivided so that one parcel does not have any way to reach a public road (is landlocked), then an easement of necessity is granted over the parcel that adjoins the land-locked parcel to allow exit and entrance. Such easements usually are recorded with the land deed and remain a part of the land thereafter.
Of Convenience - this easement grants the right of ingress and egress to a parcel of land for the convenience of the land owner. It is generally given to the owner of a parcel who sells a part of that parcel to someone else. If this is the case in your situation, then your neighbor
's land was the parcel that was split and your lot was the one sold off and set to "serve" the other lot by allowing ingress and egress.
Granted - it is possible that a former owner of your land voluntarily granted an easement to your neighbor to permit him to enter and leave his lot by using your lot. Such grant may have been recorded with the deed and become a part of the land or may have only been granted to the person and not on the land. If only given to the person then you may have the rigth to terminate your current neighbor's use of the easement by filing a civil case against him in court. BUT, constant use of that easement may make it permanent. Only an attorney can tell you if you can argue to terminate the easement.
The land belongs to you, but, you cannot deny the easement owner the rigth to use the property if the proper record of the easement or time has granted him the easement. Real estate
law is very complicated, however, and in cases like this requires not only a property search of you land records but those of your neighbors to determine what the rights are. If you are having difficulties with the easement owner you need to speak to an attorney who can peform the research and provide you with an explanation of the actions you can take.