I'm sorry, but this is not likely to be something that can be resolved, if at all, through any channels between now and tomorrow other than you talking it out with the birth mom.
As part of an adoption there is a complete and total termination (a severance) of all parental rights of the birth parents, and all of those rights are transferred to the adoptive parents. The birth parents are strongly reminded at the court proceeding (if they are participating voluntarily) that once the court hands down its orders they will have NO parental rights of any kind. The intent is to make sure that their expectations match up to reality and that they harbor no reasonable hope of having any parental relationship restored.
However, you have re-bridged that chasm, restored hopes and expectations, and re-granted some form of parental rights to the birth mother through the "temporary guardianship document" that you executed on your own, and through the process of giving the child back to her to live with temporarily. In that process, you have also moved outside of the field of what the law would require of the parties.
In the long run, if the birth mother declines to return the child to you, but asserts some rights in the child by virtue of the fact that you have given her temporary guardianship and you have put the child in her custody and care, it will be a matter that ends up having to be reviewed and decided on by the court. It is one of the court's jobs to sometimes have to play at being King Solomon and somehow form a resolution in a case of conflicting and impossible expectations on both sides.
It may be that the court will rule all is well for you and that the temporary guardianship role was yours to grant and yours to rescind at any time. However, I can't assure you of that. They could also rule that you have restored a significant parental relationship with the birth mother toward her former child, and that once this Pandora's Box has been opened, their relationship has to be honored in some form.
As for the immediate situation, your choices are limited as far as I can see: you can continue to try to talk the birth mom into letting you take the child for this trip; you can take the trip without the child (and tell the people you are visiting that you left her behind with a sitter); or you can skip the trip. However I can see nothing that would allow you to call the law into the equation to lend support to your side. (For instance, if you were to call in the sheriff and ask them to take the child from the birth mom and return her to you, they would throw up their hands, state that there is no clear cut basis for knowing who has the right to the custody of the child in this circumstance, and tell you to resolve the situation in family court.)
I wish I had better answers for you, but you have taken this situation more or less off the map as far as the legal processes and procedures go. There is no blueprint for exactly what you can do next.
The information provided is general in nature only and should not be construed as legal advice or to create an attorney-client relationship. You should always consult with a lawyer in your state.
PS: If an answer appears to you to have been very helpful, or to have taken above average expertise and/or research, or if the answer shows an above average amount of time and dedication devoted to your issue, a bonus is nice way to say "Thank you". Thanks!