Your question was "is there any way the IRS can come after you", and unfortuantely, the annswer is "YES", they can. Since your husband is a resident of CA and you are a resident of LA, these are both Community Property states. That means the debt of one spouse is also the responsibility of the other, UNLESS you can show that you had separate funds and accounts.
What you need to do, is to file an INNOCENT SPOUSE claim form with the IRS. This is different than INJURED SPOUSE. An Innocent Spouse is someone who is unfairly tagged with the tax liability of a spouse or former spouse because they had no knowledge of the tax return and were not involved with the income or the return preparation. This is different than INJURED spouse.
Click this link to the IRS webpage for Innocent Spouse claims