I will opt out so that another expert may be able to answer that question.
What Michelle has told you so far regarding NYS income tax is correct. The state deems your income as "New York Source" income because your business or trade originates from the state. You have not mentioned if you are an employee or self-employed, if self-employed, you may be able to allocate based on volume. If an employee, then it is most likely that the state will insist on taxing entire income as NY source income.
You may wish to view this article, http://www.tennessean.com/business/archives/05/03/67569995.shtml?Element_ID=67569995 this was a recent case that established NY's "authority" to collect tax on income from a person who did most of his work in Tennessee. I believe that your case is similar.
If after reading this article and instructions in NY non-resident form on pages 19 and 55 that address this issue, you are still in disagreement with the assessment of tax on your entire income, then you should hire a tax professional to assist you in organizing your argument. http://www.tax.state.ny.us/pdf/2007/inc/it203i_2007.pdf
Keep in mind, you can apply for a PA credit for tax paid to other jurisdiction for taxes paid to NY.
I'm sorry we cannot be of more assistance to you. I have not personally been involved in NY audit process, but from what I have seen others go through, I agree that NY's attempts at collection are adversarial and difficult. With money like that at stake, I would definitely hire an experienced party to represent you.
You may also wish to contact PA department of revenue and see if there is a procedure or a way to ask for special permission to extend the 3 year limitation on refunds. You definitely have a basis for this. If not, you may amend your 2004 PA form now with the presumption that you are due a refund based on credit for tax paid to another jurisdication because of taxes due to NY.