There is no provision allowing you to rollover your Canadian RSP or RRSP to a U.S. IRA. Pursuant to the provisions of the U.S.-Canadian tax treaty your distribution from the RSP (or RRSP) will generally be taxable by both Canada and the U.S. For Canadian tax purposes, the tax is limited to 15% of the gross distribution. For U.S. tax purposes the amount taxable by the U.S. is limited to the amount that a Canadian resident would have to report to Canada. You may want to review IRS Pub 597 which provides information about the U.S-Canadian tax treaty at http://www.irs.gov/pub/irs-pdf/p597.pdf. Accordingly, you will likely need to file a Canadian Non-Resident tax return in order to get a refund of the excess withholdings (i.e. tax withheld at 25% but taxed at 15%). I suggest you talk with a Canadian Chartered Accountant or a U.S. CPA who is familiar with Canadian taxes.
Generally, only income taxes paid or accrued to a foreign country or a U.S. possession, or taxes paid or accrued to a foreign country or U.S. possession in lieu of an income tax, will qualify for the foreign tax credit. Qualified foreign taxes do not include taxes that are refundable to you or taxes paid to countries whose government is not recognized by the United States.
You can choose to take the amount of any qualified foreign taxes paid or accrued during the year as a foreign tax credit or as an itemized deduction. To choose the deduction, you must itemize deductions on Form 1040, Schedule A. To choose the foreign tax credit you generally must complete Form 1116 and attach it to your Form 1040 (http://www.irs.gov/pub/irs-pdf/i1116.pdf).
The amount of the foreign tax credit will be the smaller of the amount of foreign tax paid or accrued, or the amount of United States tax attributable to your foreign source income. This limit is computed separately for each type of foreign income.
If you cannot use the full amount of qualified foreign taxes paid or accrued, you may be allowed a 2–year carryback and then a 5–year carryover of the unused foreign tax.
You may not take either a credit or a deduction for taxes paid or accrued on income you exclude under the foreign earned income exclusion or the foreign housing exclusion.
I assume you do not reside in Canada nor are you there for more then 330 days over any 12 month period. If you are this may change the answer to your question.
With respect to the foreign tax credit you may want to chack out IRS Pub 514 at http://www.irs.gov/publications/p514/index.html.
Because it is impossible for me to identify and consider ALL the relevant facts, this advice is not intended or written to be used for the purpose of avoiding penalties, and cannot be used for that purpose.