Get UK Law Questions Answered by Verified Experts
S3A of the Race Relations Act 1976 defines racial harassment as where on the grounds of race he engages in unwanted conduct which has the purpose or effect of (1) violating that other person's dignity or (2) creating an intimidating, hostile, degrading, humiliating or offensive environment for him.
Unfortunately there is no exhaustive list of what is racist and what is not. The question of whether the statement itself is inherently racist is one over which lawyers have no more moral authority than other members of the public in my opinion. Racists have adopted and used it, so you would have this hurdle to overcome if you were to argue that it was a statement about naivety.
For what it's worth my Pakistani colleague doesn't think it is a racist statement. In any event it appears that you have reacted in a sensitive fashion.
If this is useful please kindly click accept so that may be rewarded for my efforts. It will be gratefully received and you will be free to ask followup questions.