In short, no, they should not be able to take your money if you do not move it soon. There are regulations and license requirements that these investment types must adhere to when conducting business that are in place. That said, you need to look at and be aware of what rights they have and penalties can be charged that should be stipulated in the contractual language to which you have agreed to in your retainer with the entity.
There can be early withdrawal penalties that are contractually owed if you mover your business. Moreover, depending on how you do it, there can be tax obligations for withdrawals from certain investment accounts. You need to have a firm grasp of all of that before you do anything.
To that end, it would be worth the effort to meet with another suitable investment specialist about turning your business over to them and have them look over your investments and information and provide you a pathway to switch and avoid as many penalties as possible.
Any reputable investment professional will be well versed in this exact scenario and should be able to give you an "escape route" with the path of least resistance.
Let me know if you have any other questions or comments.
Please rate my answer positively as well.
Best wishes going forward!