Whether you are an agency worker or not, your employer is not permitted to change the key terms of your contract without your agreement during the term of your employment. The nature of the work that you are doing is certainly a key term. If the contract does not state what the work is, then there would be an implied term at this stage that whatever work you have been doing all along is what you are employed to do... that is, unless there is a term in the contract dealing with the fact that the nature of the work might change as needed etc.
I wonder why the nature of your work is changing. Is it a personal matter, specific to you; in other words are they picking on you for some reason? If that is the case, then perhaps discrimination or bullying etc are relevant to your situation.
Or, is the work that you are doing no longer necessary, so that your post is effectively redundant? If that is the case, then the employer is no longer obliged to keep you on, but perhaps they do not want to make you redundant.
So, the short answer is that your employer can't change the terms of your contract during the currency of the contract (unless the contract itself provides for a very wide range of duties to be performed by you as required). On the other hand, the whole circumstances will be relevant as well in terms of whether you can or should do anything about it. On the basis of what you say, I would suggest speaking to the agency about your concerns to begin with.