You are not a horrible mother. If you are feeling guilty, your behavior even subconsciously displayed may be picked up by him.
You were doing what you had to do at that time in your life. Children are resilient and can also change and learn new behaviors as time goes by.
Patience would be needed on your end even if he is resistant. You are doing what you ought to be doing with him- spending time, playing, etc. He will get used to the new activities and your presence with time. Continue to engage him in the activities that he finds pleasant/fun.
Thank you so much for your feedback. I think maybe he is picking up on how I feel about the situation, which is extreme guilt and rejection. So I guess it is just my issue? I certainly hope so. I would hate for him to grow up with unresolved subconscious rage directed at me. I think ultimately that is what I am afraid of. Then again, I am completely projecting my feelings towards my own mother onto him then. It is hard to know the difference between what my issues are, what is normal for a baby, and what isn't a big deal at all. So are you saying that there isn't an attachment issue? It is just change and getting used to things as they change?