If you do want to be with him then the best way is to explain everything too him. Help him understand that the expenses to get there is something you are unable to do right now. That when he gets out you will be able to see each other all the time. Explain that you are here for him and that you are not going anywhere. That there are other ways for you to communicate and create a strong relationship. Through letters, phone calls, and that you will try to come visit, but it can't be all the time. I think he got scared because you were not visiting that you were leaving him. It's hard because he has no control of anything in his life right now and he relying on you to make life a little easier in there and visits are very important. But he also needs to be understanding that it takes money to get there and if you don't have it, he has to be more understanding of your situations. But that you will write and that when he is out you both can start a life together.