Unfortunately the information you were given was not correct. As a US citizen, if you live and work abroad for a period of at least 330 days in a 12 month consecutive period, you qualify for the "foreign income exclusion". That exclusion amount is currently set at $85,700. In prior years, the amount was probably somewhat lower than the current amount.
If your foreign income did not exceed this amount and if you had no income from other sources such as investments, then in all likelihood you would not have owed any US taxes, however, you were still required to file a return to report your earnings and claim the exclusion.
Since you have all of your earnings records, then the best thing for you to do at this point is to try and file the returns for the years you missed. You may need the help of a tax preparer in doing this, or you may also obtain help free of charge by calling the IRS. They will be able to help you get the forms you need for the years that you need to file. I am also giving you below a link to a section of the IRS website that explains the foreign income exclusion and gives you links to the forms you need for the current tax year.
If it ends up that there are any years where you do owe tax, then in addition to the tax owed you will no doubt have penalties and interest for the late filing and payment. However, penalties will not apply in any year where there was no tax liability. So if your foreign earned income did not exceed the exclusion amount in any given year, then hopefully you will not have to worry about having any taxes or penalties to pay. It will, however, take you a little time and effort to get these returns filed, but then you will be up to date and can start filing your returns each year as they are due.
If this was helpful, please press the Accept button. Positive feedback is also appreciated.