It appears you went offline without seeing my request for additional information. I don't want you to have to wait any longer, so I'm going to go ahead and answer without knowing that. Both types of pepper trees do cause problems. The California pepper tree is taller, so I do suspect that's what you have. The Brazilian pepper tree often grows as a large shrub (15 to 30 feet tall) with multiple trunks. It sends up suckers. The California pepper tree has an especially shallow root system. The roots are often invasive, working their way under foundations, sidewalks, patios, and streets. This results in cracking and more serious damage to structures. The trees are also prone to losing branches, and the female trees produce messy berries twice a year. Both types are considered invasive, either by seed dispersal or shoots coming up. For all these reasons, some cities have made it illegal to plant pepper trees.
The California pepper tree is adapted to growing on its shallow roots in open spaces. On urban lots, there may not be enough space for those roots to spread out and anchor the tree. When that happens, the tree may topple in wind. So, you'll need to look at your situation. If there's plenty of room for the roots to grow without running into any structures, they may be sturdy enough to hold up the tree. If not, it may be best to remove it.
If you have more questions, let me know in a REPLY. I hope that whatever you decide to do, it will work out well.
My goal is to provide you with excellent service – if you feel you have gotten anything less, please reply back, I am happy to address follow-up questions. Please remember to rate my service only after you have all the information you need. Thank you!