Okay, I am going to have to give you an answer that is unfavorable, so kindly do not blame the messenger if the answer is not as you hoped. This answer is based on US employment law, which is not as potentially progressive as UK law or other European nations.
The reason I asked about housing is because the employer in the US is generally not responsible as to what happens to you outside their physical location. An argument can be made that they share some liability if they provided housing. In a situation where you found housing yourself, you are personally responsible for that decision, which means that you cannot use this incident as a legitimate basis for terminating your obligation to your employer. The employer has a right to hold you to the terms, and unless the contract expressly has a clause against assaults or threats off-site or other reasons for termination, you are bound by the agreement. Likewise if the agreement expressly lists a set amount for 'liquidated damages', and you breach, you are responsible for such a fee unless you can prove and show that the amount is 'unconscionable' and extremely unreasonable. That I cannot answer as far as a good argument purely because I do not know how much training or expenses the employer put forth toward you, as that is really the basis under which they figure out costs and whether or not the liquidated damage provision is reasonable.
Likewise an employer under US law has a duty to provide you with 'reasonable accommodations' only in cases of disability or impairment. As this took place off-site, the argument that you were unsafe would not work because the work environment itself is safe. In addition, even under the employer's duty, the obligation is for a 'reasonable' accommodation, which means that if the employer cannot justify moving you elsewhere as it would be too expensive or otherwise unreasonable, they are not in violation of law.
This therefore leaves you with three choices. You can attempt to negotiate and have them agree to waive that clause. This is unlikely but it can be tried. You can continue to work as you are doing now. Or you can quit but then be prepared to pay the amount back based on the language under contract. Under this option you can elect to go to court and try to fight it but my concern is that you may still lose and you would have additional attorney fees as expenses as well.