S3A of the Race Relations Act 1976 defines racial harassment as where on the grounds of race he engages in unwanted conduct which has the purpose or effect of (1) violating that other person's dignity or (2) creating an intimidating, hostile, degrading, humiliating or offensive environment for him.
Unfortunately there is no exhaustive list of what is racist and what is not. The question of whether the statement itself is inherently racist is one over which lawyers have no more moral authority than other members of the public in my opinion. Racists have adopted and used it, so you would have this hurdle to overcome if you were to argue that it was a statement about naivety.
For what it's worth my Pakistani colleague doesn't think it is a racist statement. In any event it appears that you have reacted in a sensitive fashion.
If this is useful please kindly click accept so that may be rewarded for my efforts. It will be gratefully received and you will be free to ask followup questions.
DISCLAIMER: Answers from Experts on JustAnswer are not substitutes for the advice of an attorney. JustAnswer is a public forum and questions and responses are not private or confidential or protected by the attorney-client privilege. The Expert above is not your attorney, and the response above is not legal advice. You should not read this response to propose specific action or address specific circumstances, but only to give you a sense of general principles of law that might affect the situation you describe. Application of these general principles to particular circumstances must be done by a lawyer who has spoken with you in confidence, learned all relevant information, and explored various options. Before acting on these general principles, you should hire a lawyer licensed to practice law in the jurisdiction to which your question pertains.
The responses above are from individual Experts, not JustAnswer. The site and services are provided “as is”. To view the verified credential of an Expert, click on the “Verified” symbol in the Expert’s profile. This site is not for emergency questions which should be directed immediately by telephone or in-person to qualified professionals. Please carefully read the Terms of Service (last updated February 8, 2012).