Of course drs aren't supposed to look down your shirt, nor look at you as anything other than a patient. It is against their rules for him to become sexually or otherwise involved with you while you are a patient, and for at least up to 6 months after you are no longer a patient (time frames can vary by state). With that being said, they are still human, and they are still men.
As women, we are attracted to doctors, after all they are the healers in our society and have an almost magical presence to us mere mortals (smile). It can be easy to want to be with such a man and have him find you attractive. However, I would tread carefully. If he is in fact coming on to you, you may want to ask yourself what other women he does this with, as it is against his rules if not his better judgment. He knows the effect he has on women and his patients.
You could easily ask him out for coffee if you're trying to see if he's interested, but keep in mind you would need to drop him as your doctor and wait the required amount of time before getting involved with him. If you decide to pursue anything, keep in mind he could lose his license if it's ever found out, and anyone from his secretary, one of your friends or a patient could report him.
It could be that someone saw it and said something to him about it.
Or it could be that he realized he was showing you favorable attention, whether it was because he liked you or not, he was still doing something that could be frowned upon, or maybe come back to him later.
As I was saying in my earlier posts, doctors have to be so careful of impropriety, anything they do can be taken the wrong way, and they are in a position to be sued or losing their license.