The mileage to/from home to your place of work, no matter how far is considered to be personal mileage.
On the days you go to the distant factory only, none of your mileage is deductible. On the days you stop at the near factory first, the mileage from the near factory to the far factory is allowed as a deduction.
The deductible mileage would be a miscellaneous itemized deduction. The total of all miscellaneous itemized deductions that exceed 2% of your adjusted gross income would be deductible.
I have provided a link to IRS Publication 463. See page 14 for details on how to determine business miles.