IWhat should I do?

I have been seeing this man for 4 years. We were neighbors living in the next building to one another until he moved February 2006. When I would leave my apartment he could see me. When he leave he his apartment I would look out my window and see him. I would keep an eye on him as much as I could. But to tell you the truth, he started this close watching thing. As soon as I would leave my apartment and he see me walking across the parking lot to go to my car, he would call me on my cell to find out where I was going. Anyway, every now and then I would see a woman go into his apartment. I did approach with this matter and he said she was just a friend and that she is there to do paper work. I certainly did not beleive him. However, the woman would come by to my knowledge twice a month and when she would visit him she would stay in his apartment for about 2 hours. It was funny, but I would never see her on week-end at his place.   Unless it was a Sunday afternoon. Again her time limit there was about 2 hours.   A couple of times I would see her come out of his apartment 8:00am. She would never stay to mid moring or the afternoon. She would always leave very early. I thought perhaps she had to go to work, until I found out where she actually live. The woman does not leave for work. Since I work not far from where she live; I would go by her place and see her car in the morning and afternoon. She never leave out until about 6:00 or so. This is so strange to me. Anyway, my former neighbor calls me 2-3 times a day. I do not know why. He can have his friend over any weekend now. She can stay over as late as he wants but he calls me early in the morning. And because I know where she lives, I would go to her place to see if she would be at his house in the wee hours. She would not be at his home. I am so confuse. My neighbor has ask me resently (last night) Why I do not call him. My real reason for not calling him is because I can't trust him and I do not want to start this relationship up again. I am not getting much from it. I have turned down 3 men for him. Three very good men! I have talk to him about this woman over and over again and have made suggestions that perhaps him and I should have an open relationship. Of cause he told me that he wants a monogamus relationship. I said ok I still continue to not trust him. He calls me too much until I cannot even start seeing anyone because he will call me and if I do not pick up my phone he will just call me over and over again until he reach me. He will call me at my job as well. This is the only major thing I have against him. I think he is deceiving me. I have not invited him to my new apartment at all. He has not invited me. But I did help him to move. I spied on him once and I saw that woman at his apartment. But I was never, never invited to just visit him. At this time I will never invite him to my place. He made a statement that I have not yet invited him to my place. As for me, I have not said a word about not being invited to his apartment.

I am having a hard time telling him to move on and leave me along.

What do you think about this situation?
What should I do?
Should I continue to talk on the phone with him?

I am confuse!!

Please Help!