Hello and thank you for your question.
A verbal agreement can still be legally binding if the parties intended it to be, however, because it is not in writing proving such an agreement can be almost impossible and even if the parties did agree terms, if it is not possible to deduce what the intended terms were meant to be, then the contract would likely be void for uncertainty. So if you hadn't negotiated all the key terms, including pay, work hours, the roles requirements at a minimum then legally it is highly doubtful you have a binding contract.
Even if you had entered into a binding contract to commence work on a particular day, by giving them notice now, it seems unlikely a court would find any damages attach to such a breach, since they appear to have ample opportunity to find a replacement.
Based on what you have said, I think it is reasonably arguable that your 'gentlement's agreement' was not intended to be legally binding, but rather to be a mere preliminary understanding, since you likely both would have expected a more formalised and perhaps written agreement before you would been willing to legally bind yourselves. So legally, I don't think you have much to worry about if you withdrawn now, provided you let them know whilst they still have time to make other arrangements.
My suggestion is that you simply write a letter to them stating that you are no longer interested in the position contemplated in your previous discussion, (perhaps adding an explanation if you are willing to share your reasons, but you don't have to).
It is most unlikely they will take legal action in such circumstances, but even if they do, I think they would have almost no prospect of suing you or trying to force you to proceed with accepting the appointment.
I trust the above assists.
Please rate my answer.