You should inform your ex that if he does not return the child you will need to take legal action. Refusing to return a child is a serious matter. He may have committed 'Parental Kidnapping". Here is the definition:
In the absence of a court order determining rights of custody or visitation to a child, a person having a right of custody of the child commits the crime of parental kidnapping if he removes, takes, detains, conceals, or entices away that child within or without the state, without good cause, and with the intent to deprive the custody right of another person or a public agency also having a custody right to that child.
"* 28 § 1738A. Full faith and credit given to child custody determinations
(a) The appropriate authorities of every State shall enforce according to its terms, and shall not modify except as provided in subsection (f) of this section, any child custody determination made consistently with the provisions of this section by a court of another State.
Remember, it is not just your baby. It is also her baby. You have no more legal or moral right to deprive him of the child than she has to deprive you of the child. You could have a court determine that you will be chosen as the custodial parent. Most states determine who will be the custodial parent by considering the best interests of the child.