Great question. For my money, it is best to let the thought run its course in silence rather than disrupt the natural flow of the present conversation. In fact, I don't think you are all that unusual, to be honest. Many of us find our minds wandering at least a little during the course of a conversation - but if we were to verbalize everything drifting through, we would be considered quite odd by everyone else!
Keep those thoughts under your hat and enjoy yourself - If you are satisfied with the response, please hit "Accept." That is the only way I can receive credit for my answer. Thanks-