It's a good question and one that is debated a lot in psychology. But in your situation, what makes me hesitate to say it is genetic or physical is the fact that you were able to live anxiety free during periods of your life.
Plus, it is quite rare for a person who has not been abused or traumatized to develop anxiety and/or panic. I can't name one person I've seen in therapy that did not have abuse or trauma that caused their anxiety. There may very well be people who develop it after a normal upbringing, but I am not aware of them.
It is hard to know what causes anxiety. But in mental health, it is not a diagnosis that is usually associated with genetic and physical causes. There are too many incidences of people developing anxiety after being in the military and after traumas such as rape. That does not happen with disease such as schizophrenia, which starts at a young age, does not seem to have a common trigger, and exists no matter what environmental or behavior issues there are for the person. Bipolar is the same.
So in my experience, I believe anxiety is caused by environmental factors including upbringing, trauma and other events that cause a reaction.