Good afternoon Leland,
I'm Doug, and I'm very sorry to hear of your situation. My goal is to provide you with excellent service today.
As the ownership of the property was not determined in the divorce, that legally meant that both your brother and his ex owned 50% each of the property at the death of his ex.
If the ex had a will, the property would have passed by will, and if nor it would have passed by intestate succession 1/3 of her share to each of her 3 children. It appears that the one-sixth of the property is presently owned by your nephew.
You may buy his share out from him, or if he will not sell, you have the option of suing him in court for Partition and ask the court to force the sale of the property in the even that it is not capable of being subdivided---which invariably single family homes are not.
Alternatively, you can go forward with your nephew as a 1/6 owner of the property and seek an arrangement where you share the rental profits as well as the cost of maintenance and taxes/insurance going forward.
You may reply back to me using the Reply link and I will be happy to continue to assist you until I am able to address your concerns, to your satisfaction.
Please be so kind as to rate my service to you. That is the only way I am credited for assisting you.
I wish you and yours the best in 2016,