I am a double board certified US doctor and I will help you with this.
You are correct to be concerned.
You should not simply stop it.
A doctor needs to help you set up a taper.
And, there are medications which can ease withdrawal symptoms.
Withdrawal alone can be dangerous and even seizures can occur.
The doctor you saw, if not concerned, was incorrect.
OK, so you might have more questions or want to give me more information: Please use reply to expert if you have further questions. Also, please click a positive rating [hopefully excellent (5)—that’s how we are paid, per rating]. If you forgot something, just use reply and come back. I am here http://www.justanswer.com/medical/expert-dr-thomas/