Since you have already attempted to reconcile via telephone, written communication is the next step; but you may wish to call back and verify what procedure is to be followed for the return of the check.
It is also possible to go to a local office and submit the information and check; but that may or may not be worth the time and effort. Locate the closest Taxpayer Assistance Center by entering your 5 digit zip code at http://www.irs.gov/app/officeLocator/index.jsp
As you may know, the CP12 Notice is produced during processing of the return after initial entry of the tax return data into the IRS system. This is a computerized process with little, or no, human interaction.
Since the computations you made on Form 6251 were based, at least in part, on information from your prior year Form 6251 it does seem as though the IRS has not taken that prior year information into account when completing the form.
From your description that adjustment of AMT basis of the options (fraction of X) is the amount that accounts for the difference between IRS and your Form 6251.
It is helpful to send a cover letter explaining the reason for the change in as simple language as possible. Including the prior year Form 6251 may also be helpful for the explanation.
Please ask if you need clarification.
My suggestion would be to number the items being sent and to reference those page numbers in the explanation. That is, I make a list of items included in the explanation letter and then use that number in my text when referencing attachments.
My tact is generally to only send (or at least only reference in the explanation) the minimum needed to convey the correct tax treatment. For that method, items 4,5 and 10 might not be sent or might be sent but as "additional supporting documents"
My goal is to make it as easy as possible for the IRS representative to see, agree and be done with what is sent.
"doing numbers crosscheck:1. Assuming I don't have any AMT adjustments other than ISOs;2. Form 6251, Line 17 (amt_net_capital_gain - regular_net_capital_gain) is a negative number."
For the 2011 AMT gain, the amount that was included for AMT purposes in 2010 is being added to the basis for AMT purposes; so the AMT gain is smaller than regular tax gain hence the difference between the AMT smaller gain and the regular larger gain is a negative number, yes.
(As only some of the options were exercised, this adjustment is only for that portion, or pro rata share, of the options sold and the balance of the prior AMT included amount will be the adjustment to AMT basis for the remaining options.
That is, using the entire prior AMT included amount for only some shares sold would give too large of a negative number)
Should Form 6251, Line 37 be equal exactly to : regular_net_capital_gain - Line 17 and be equal to amt_net_capital_gain?
As already discussed, the AMT net capital gain is less than the regular net capital gain by the amount of Line 17 so "regular_net_capital_gain - Line 17 and be equal to amt_net_capital_gain" is a correct statement.
From the Form 6251 Instructions at http://www.irs.gov/instructions/i6251/ch02.html#d0e1486 :
"Use amounts from Schedule D or the AMT Schedule D, whichever applies, and either the AMT Qualified Dividends and Capital Gain Tax Worksheet or the AMT Schedule D Tax Worksheet, whichever applies, to complete lines 37, 38, and 39 of Form 6251."
So, the AMT amounts are properly used for lines 37,38 and 39 when the gain or loss from any transaction reported on Form 8949 or Schedule D is different for the AMT than for regular tax.
Please continue to ask if you need more help