I wouldn't say that you are crazy, but it may be asking too much of someone who is married. It's not uncommon for people to look back on high school or college loves and become reminiscent and sometimes even obsessed with what they once had, or why things didn't work out, or what they could have done different or what the other person was thinking back then. If he didn't expound on why he didn't come back, he may not have had the same feelings, or he may no longer have those feelings, or even be embarrassed that he didn't come back or that he disappointed you somehow. I'm not sure what kind of contact you've had (email, phone, letters) but if he's not also reciprocating these feelings of wanting to reminisce or reaching out to you on his own, then you might want to leave it to casual talk or emails. Forty years is a long time, and people change. There's nothing wrong with being friends, or keeping in touch to a limit, but getting to focused on him or old times isn't good. I welcome your thoughts, let me know if you want to talk more.