You need to take a look at the contract to determine what your obligations to the builder are. The contract terms are going to govern. Usually, a real estate construction contract states that the contract is contingent upon the buyer securing financing. However, if the contract was written by the builder, it may not have that general language. Also, there may be a clause in the contract that states if financing fails, or if the contract is otherwise breached, the builder is entitled to liquidated damages, i.e. the $2,500.00.
The law allows people to contract on any terms, so long as they are not illegal. However, if the contract was written by the contractor, and unilateral in nature (meaning that the contractor had all of the bargining power), and some of the terms are unconscionable (no one in their right mind would agree to the terms), you may have some grounds to challenge the validity of the contract.
However, you are obligated to read a contract before you sign it, and if you fail to do so, you proceed at your own peril.
The best advice I can give you is to review the contract and see if you can find any statements regarding damages the contractor is entitled to if a breach occurs. If the language is not in there, then the contractor will have a hard time arguing it is entitled to the money. If it is in there, the next thing is to consider whether you have any options to get out from under the contract. I would suggest hiring an attorney to review the contract with you if things go this far.
Please feel free to ask any follow-ups.