You pose an interesting question. . The refractory period (in men) is the time that takes him to be aroused and or have another orgasm after the first. Your ejaculation does seems to be on a 24 hour cycle, but your arousal is not abnormal. And you are correct, that is a delayed timeframe, somewhat. The average time to repeat ejaculation is about an hour, and does advance with age.
When a man orgasms, the hormone prolactin is released. Studies have found that the levels of prolactin in males are very tied with the refractory periodOne study showed that a male without ANY prolactin had NO refractory period. It is well known that hyperprolactemia (too much prolactin) inhibits the sexual drive. It is possible that you have higher levels of this hormone than is normal for men.
This is the link to the study I cited above. http://www.nature.com/ijir/journal/v14/n2/full/3900823a.html
Here is a similar study by the same group of authors which also showed the same things: http://joe.endocrinology-journals.org/cgi/reprint/177/1/57.pdf
Unfortunately, the authors of the two studies did not choose to coorelate prolactin levels and refractory times, but it is well known that hyperprolactemia can cause sexual disfunction in men.
Other reports state that a man can have orgasms without ejaculation. Here is a link to this information: http://www.whitelotuseast.com/MultipleOrgasm.htm
My advice would be to go to a urologist for a check up. Discuss the studies I mentioned, and have prolactin levels checked.
My other thought on this would be that as you are able to get an erection, and it does feel good, can you enjoy the sex without the orgasm? I imagine it would get frustrating after a while, especially if you are under anxiety to perform. But if you did not have performance anxiety, could you enjoy yourself? Could frustration with not being able to ejaculate stop you from ejaculation?
I hope I was able to help,