It's usually better for the dog and for you if you just make the switch and be done with it. I would not even plan to visit until the dog has been with the family for at least 3 months and fallen in love with his new people.
If you want to visit the home with the dog, it's a good idea to spend an hour or so there so that you can see how he interacts with his new family. Then take him back home. Don't leave him for a while and then go back and get him. This will make him look for you even after you have left him permanently.
Let me know if you still have questions about this.