Thank you for your follow up.
Basically, until the legal custody / visitation is established by the court, both parents have equal rights to the custody of the child and if the child is currently in her father's custody, it would not be possible to get her removed from the father's custody without the court order, unless the father is causing harm to your daughter or pose a risk of potential immediate harm.
So basically, either parent can file a Petition for Child Custody
Visitation and it does appear that your boyfriend might be misinforming you about child custody laws and manipulating you, so I would strongly urge you to immediately consult and retain an experienced local experienced child custody attorney to represent you in your upcoming child custody proceedings.
Even if you go to the mediation and agree on the child custody / visitation, such agreement would still have to be filed with the court and the court would have to review, approve and order such child custody / visitation agreement, before it becomes legally binding and enforceable.
I wish you the best of luck!