Your nephew, because of his age, would only qualify under the Qualifying Relative rules for dependency.
He could not have made over $4050 for the year,you would have needed to provide more than 50% of his support and would generally have needed to live with you ALL year.
There is an exception to living with you for the entire year. The person must either have lived with you for the entire year as a member of the household (a person who is not actually related to you may meet the requirements in this way), or be related to you in one of the following ways:
your child, stepchild, grandchild or other descendant of one of your children (or stepchildren or foster children), son-in-law, daughter-in-law, brother, sister, half brother, half sister, stepbrother, stepsister, brother-in-law, sister-in-law, parent, stepfather, stepmother, father-in-law, mother-in-law, grandparent, and, if related by blood, aunt, uncle, niece, or nephew.
If this is a blood relative nephew (the child of your brother or sister) and you paid more for his support than the support provided for him while he was in jail then yes, you can claim him.
You would have to add up all the support you provided, find out how much is paid for inmates where he is/was incarcerated and see who paid more.