Hello, I'd like to help you with your question.
You mentioned some very important points. One, you felt lonely when you were in the relationship with your ex. THat indicates you were not getting your needs met somehow. Whether that was because he kept you at a distance emotionally or that the relationship did
not fulfill your needs completely is not clear. However, it is important to realize that being with him was not making you happy.
The second point you made is that although the relationship is over, your ex has never accepted that you left the relationship. He keeps trying to reconnect with you but on his terms. And he is touching you and sitting close to you even while in relationships with other people. That indicates a boundary issue and a disrespect for others' feelings.
The third point was that you fear expressing your feelings for him because you are concerned that he would refuse to be with you again to "punish" you for breaking up with him in the first place.
While some of these things are subtle, they do indicate imbalance in your relationship. It sounds like he as a lot of the control and does not respect your feelings or needs. That could be part of the reason why the two of you are still connected. Sometimes in relationships that are imbalanced, people are connected by a dysfunctional link that makes them feel they "fit" with one another because they both play a part in keeping those feelings going, even if they are not healthy for either person.
What you can do is decide whether or not you really feel this relationship is worth considering again. If you are with him, will you be happy? Will you feel complete? If not then it might be time to move on. If you do decide to move on, then it is important that you cut all ties. It will be difficult because you might experience grief over the loss, but if you continue as you are now, you will never feel happy or settled.
I hope this has helped you,