To address your question if you're wasting your time with him will depend on what your expectations are.
1/ something i the past alerted you to separate
2/ he appears secretive of your current relationship
3/ he did not respect your wish not to call you
4/ he had not given you a definite explanation about his behavior
You really would need some straight forward answer from him (unless you can look at his actions and deduce what is going on)
If your expectations for the relationship do not match his, things may not work out.