Welcome to JustAnswer.
Most retail stores over the last several years have began using POS (Point of Sale) software to track purchases, returns, etc. If you asked about Walmart, Target, etc I would say their is a 100% chance they do. I cannot say about your local Goodwill stores, but I would say the chances are very high. Even services that let you take payments from a smartphone or iPad incorporate POS software of differing levels of sophistication.
So to answer your questions:
Is there a way in which, for example, Goodwill's retail stores can track suspicious returns?
I cannot say for sure with your particular stores, but probably so.
Is there a way to determine if loss prevention, etc... officials are tracking MY return activity?
No, there is no way to know. If your suspected of shoplifting, occasionally you may notice a loss prevention worker following you around the store, but with fraudulent returns they will likely just wait for you to do it again before taking any action or turn their information over to the police.
And if they are tracking your returns and have reason to suspect you are engaging in fraudulent activity, is there some way they would let you know before they pressed charges (if they do press charges?)
No, they want to catch people who are defrauding them. Why would they give you a heads up?
How do you know if you're being investigated without coming out and asking (which would be self-incriminating) let's say the store manager?
You do not until it's too late. That's the point of an investigation
. They will gather all the information they can without alerting you. If you are engaging in fraudulent activities, I must advise you to quit. The longer it goes on the chances of getting caught increase greatly.