Hi - my name is XXXXX XXXXX I'm a Real Estate
litigation attorney here to assist you.
Unfortunately, there's nothing you can do about this if you signed the loan papers and received a notice that the lender can do this. The loan papers govern the agreement, not what someone verbally told you (there's likely a document in the loan papers that says this as well - that the documents govern the loan and is the complete agreement between you, and that you can't rely on any statements made which are contrary to the loan documents).
If you signed the loan documents, you've already agreed to this, and you can't back out now. Also, if you didn't read the documents (most people don't) that's not a defense.
That said, there's nothing you can do about the loan being sold as long as the terms of your loan don't change - which shouldn't happen.
I do certainly understand your concern, but if things go south, some other lender would purchase the note and you'd pay that lender under the same terms.