Your mortgage payment seems excessively high considering you are paying 4.5%. I will have to assume you are paying a high amount in principal each month for it to be that high. The interest portion should be around $656 per month. Perhaps you could refinance, if only to stretch the loan out for a longer payment period. You should be able to lower the payment significantly.
If you decide to pay off the mortgage, the taxes and penalties will be a least 43%. This means you would need to take out a gross distribution of $307,000. I agree with your financial adviser. Since this is your only retirement asset, it should be preserved, at least until you are 59 1/2. Then you won't have the 10% penalty.
Thank you for your question. Let me know if I can help you further.