As a self employed individual, mileage is only deemed to be business is you travel to several sites within one day, or every few days, neither which seem to apply here.
So the normal rules will not apply to you, (which can consider temporary workplace) as your pattern of normal travel is several sites over a course of a year, so unless you leave one site to travel to another in a typical day, will any business mileage be due to you. (Or visit a different site every day or every few days)
I am sorry the news is not better, but any claim you make will be revoked by HMRC.
Sorry forgot to mention im registered cis ( construction industry scheme )
The response is still the same, (as you are treated as self employed when a subcontractor within the Construction Industry)
The answer does not change - and I would suggest that the claims being made by those you know, will be revoked if HMRC make an enquiry into their affairs.
Self Assessment sees HMRC accept the figures as claimed on the tax return, but random (and selected) checks are carried out each year, and if one of their tax returns is selected and the mileage asked after, then as their work is the same as yours, their claim will be revoked on the basis that this mileage is deemed to be normal work pattern.
I fully understand that you are self employed, as a sub contractor within the Construction Industry, and that you are engaged by a Contractor.
I worked for HMRC for 26 years, working self employment (and employment and property income and corporation tax) and during the latter year was providing educational workshops which extensively covered what constituted business mileage (amongst all other aspects) for self employed and CIS subcontractor and contractors, and I can advise if there was any way for you to make a valid claim, I would offer this to you.