It is not unheard of that even minimal cataracts can cause diplopia. It is really more about the location of the lens opacity and not the density of the cataract.
Also, corneal topography is not a test normally performed as part of a complete exam. You may want to obtain this test in order to unequivocally rule out irregular astigmatism as a possible cause for your symptoms.
Good luck as you search for a solution to this problem...
This possibility does exist; however, it is so rare that it is not documented in most ophthalmology texts. If you have been told this, I would encourage you to be examined by a neuroophthalmologist. These are specialists in just this type of situation and certainly there is no substitute for a complete exam by one of these specialists. He or she may decide further management that my include radiology testing and a full neurological evaluation by a neurologist.