Thank you for that additional info. The best thing is that he's already calling you, so that indicates that there is some interest there. Being that he felt a little slighted in the past, be careful how you approach things, because he may still feel defensive. Now that you feel like you want to get closer to him, he may feel like he has the upper hand, so you want to sit back a little, allow him to contact you, don't always answer his calls, sometimes let it go to the machine and call him back later. If you're on the phone with him, make a point of being the first to get off the phone first. These will keep him on his toes and not let him feel like he's controlling the situation.
Once you do start hanging out, just be yourself. It's obvious he likes you for who you are, so don't try to impress him or do anything you wouldn't normally do. Ask him about himself, show interest in him as a person and the things he likes. It's as important to listen as much as you talk. When he talks, ask questions about what he's talking about.
I'm not sure if you've been intimate with him, but holding off on that as long as you can is important. At least until you have an idea of where he wants it to go. Unless you're not interested in a relationship, then by all means, do what you feel. I welcome your thoughts, let me know if you want to talk more.