Generally speaking (I say generally since in the US, each state has their own system and there is a central, federal system...and the rules can vary slightly between systems) there are two ways for a judge to be removed after appointment.
1. The judge steps down. Again, this is commonly referred to as "recusal"
2. The case is presented to an appeals court and the appeals court rules that there was actual or implied bias and sends the case back for a new judge.
So to answer the question
Is it Only the appeals court that removes a Judge in US?
IF the judge is questioned about potential bias from the parties in the case and challenged (asked to step down) and they refuse? Then it is up to the appeals court to remove them.
And this leads to your next question
Are there any other circumstances when a Judge is, can or should be removed, besides removing themselves
The circumstance would be bias. Either actual (the judge has a bias in the case...example, say that the parties are litigating if A should pay B on a contract claim. And say B is the spouse of the judge and has had detailed conversations with the judge about their side of the story and the judge has confided in B and others that they support B's position. This would be "actual bias" (since the judge admitted to a position on this matter).
But actual bias can be VERY tough to prove...since its not yet possible to read minds, tough to know if a judge is biased.
Thus springs the legal concept of "implied bias"
Perhaps a good way to illustrate implied bias is to look at a state statute that details what it means. From WA state:
A challenge for implied bias may be taken for any or all of the following causes, and not otherwise:
(1) Consanguinity or affinity within the fourth degree to either party.
(2) Standing in the relation of guardian and ward, attorney and client, master and servant or landlord and tenant, to a party; or being a member of the family of, or a partner in business with, or in the employment for wages, of a party, or being surety or bail in the action called for trial, or otherwise, for a party.
(3) Having served as a juror on a previous trial in the same action, or in another action between the same parties for the same cause of action, or in a criminal action by the state against either party, upon substantially the same facts or transaction.
(4) Interest on the part of the juror in the event of the action, or the principal question involved therein, excepting always, the interest of the juror as a member or citizen of the county or municipal corporation.
As you can see, each of the 4 prongs of the test show a relationship that if the judge had with one of the parties would lead a reasonable person to believe they are biased and can not be a fair judge in the case
Please let me know if you have more questions. I am happy to help if I can. Otherwise, please rate the answer so I may get credit for my work.