What you have done is in the past. What is most important know is what you want to do hence forth?
A first love is hard to forger and it leaves an imprint whether subconscious like in your case (you are saying you've just recently realized that there is no one else that you want to be with) That can be an anxiety provoking situation.
It is unclear from the post if he is in a relationship himself with someone else (or if he had been married, had kids, etc) because in time all of these factors will come into play in your own situation should you and him decide to pursue a relationship.
You may want to engage in some soul searching and inner healing before any kind of a drastic change takes places in your life. Perhaps ask yourself the question of what will happen down the road if things don't work out?
It will be beneficial to consult with a relationship coach to help you clear your mind and sort your feelings out.