Your sister-in-law may not sue your mother for the mother's refund - that refund belonged to your mother - she may only sue for damages resulted by improper actions.
If your mother lawfully claim a child as a dependent - there i s no improper action from her side.
If your sister-in-law thinks that she is entitled to a refund - she should file a tax return and request a refund from the IRS.
Please see the IRS publication 501 for rules for claiming a dependent - http://www.irs.gov/pub/irs-prior/p501--2005.pdf
In additional - your sister-in-law should consider the statute of limitation to claim a refund - that is generally three years from the date the refund was due.
Thus - 2005 tax refund was due on Apr 15, 2006 and the statute of limitation run out of Apr 15, 2009 - that means - if your sister-in-law decides now to claim her refund from the IRS - she will not get any refund even if it was due.
While I think that chances for your sister-in-law to win are slim - your mother should be prepared for the court.
Let me know if you need any help.