Please see for reference the IRS publication 590 - http://www.irs.gov/pub/irs-pdf/p590.pdf - page 45 - Prohibited Transactions
Generally, a prohibited transaction is any improper use of your traditional IRA account or annuity by you, your beneficiary, or any disqualified person. Disqualified persons include your fiduciary and members of your family (spouse, ancestor, lineal descendant, and any spouse of a lineal descendant). The following are examples of prohibited transactions with a traditional IRA. -- Borrowing money from it. -- Selling property to it. -- Receiving unreasonable compensation for managing it. -- Using it as security for a loan. -- Buying property for personal use (present or future) with IRA funds.
So far - you can't borrow the money from the IRA account. If you take the money to make a down payment - that would be a distribution and full amount of distribution will be generally your taxable income. Additional 10% early distribution penalties will apply if you are less than 59 1/2.