Hello and welcome to JustAnswer. Please note: This is general information for educational purposes only and is not legal advice. No specific course of action is proposed herein, and no attorney-client relationship or privilege is formed by speaking to an expert on this site. By continuing, you confirm that you understand and agree to these terms.
I am afraid yes - there will always be challenges as long as you maintain this lifestyle, be it for work or pleasure.
As a greencard holder, you are expected to live in USA. This means spending majority of your time out of the year in USA, maintaining a living residence in USA, and not traveling outside USA for more than 6 months without a permit (I-131).
You have been maintaining some semblance of residence in USA and have been back coming for a few weeks at a time. However, you know as well as I that this is superficial. While highly subjective, sooner or later ICE/CBP/USCIS is going to bar you from entry into USA and revoke your greencard based on "abandonment."
To avoid this, you want to:
1) Become a US Citizen (no travel restrictions then)
2) Stay in USA for more time that you are abroad, or
3) Use an I-131 permit every time you go out for 6+ months.
Otherwise, you may be denied entry one day and yes the authorities have this discretion - they do not need a court order.
Please note: If I tell you simply what you wish to hear, this would be unfair to you. I need to be honest with you and sometimes this means providing information that is not optimal. Negative ratings are reserved for experts who are rude or for erroneous information. Please rate me on the quality of my information; do not punish me for my honesty.
I hope this helps and clarifies. Please use the SEND or REPLY button to keep chatting, or please RATE when finished. You may always ask follow ups at no charge after rating. Kindly rate my answer as one of TOP THREE FACES/STARS and then SUBMIT, as this is how experts get credit for our time. Rating my answer the bottom two faces/stars (or failing to submit the rating) does not give me credit and reflects poorly on me, even if my answer is correct. I work very hard to formulate an informative and honest answer for you; please reciprocate my good faith with a positive rating.