I was hoping to hear back from you, but I don't want you to have to wait any longer for an answer, so I'm sending my answer now,and if you have any additional concerns, please just click reply and I will be glad to continue.
From your description, it sounds like your cat is feral and once feral, always feral. You have made great strides with him in his allowing you to pet his back and being talkative with you, plus, his rubbing against your legs is showing affection. Many cats don't like to be picked up at all, but especially feral cats. Cats feel most secure with all four paws on solid ground.
He trusts you to a degree, now that you've had him for a year, but he doesn't trust strangers (visitors) that come to the house, so that's why he's stays away until they're gone.
You've made great progress with him in the 1 year that you have him, and this will be a gradual process for you both, so keep on giving him loving, talking to him constantly and try to engage him in some play sessions with interactive toys like a laser pointer and feather wand and also the types of toys listed below (there's a dvd he should love!):
Also, get some Feliway plug ins. These emit cat calming pheromones and will help him feel more secure and less timid. Here is more information: http://www.feliway.com/us
As time goes on, he will begin to trust you more and allow more physical contact, but he might never let you pick him up and hold him; don't take this personally--he obviously adores you, he just might not feel safe enough off the ground.
Positive reinforcement is great! Every time he allows you to pet him and gets close to you, physically, praise him and give him a cat-healthy treat.
I hope this helped and things will continue to progress well, with your furry boy!
Please be so kind as to rate my answer with positive feedback; that is the only way I receive credit for my assistance. Thank you very much!
Providing a positive rating will not end our conversation, should you need me for any follow-up. Simply click 'Reply' for clarification or additional information, if needed, at no additional charge.