Your first option is to try and work things out amicably with your son's father with respectful conversations. It sounds like...from just what you have briefly mentioned...that your son's father may give you some resistance. In order to avoid any miscommunication as to when visitations are to occur, you should obtain the advice of a family law attorney, go to court, and have everything written out as to what should be occurring such as the visitation schedule. That way, your son's father must abide by court appointed orders. This is a very touchy situation but keep the best interest of your son in the forefront.
Hope this has helped. PLease ACCEPT.