Hi, I'd like to help you with your question.
It is probably a good idea to let your husband know. Although that would be very hard, the truth is always better. Keeping this to yourself helps keep your attraction to your colleague going because right now there are no consequences to your, or his, behavior. Also, you need to consider how keeping such a secret impacts your ability to be close and loving to your husband. And you also need to think about your husband finding out on his own as opposed to you telling him. If he should be told this by someone else or find out otherwise, you will not only have to deal with the affair with him, but also the fact that you kept it secret. It will be very hard to come back from something like that as opposed to you starting this off by coming clean and being honest.
If you do tell your husband, make sure you have the time to talk about it. Have someone watch your baby for an afternoon so you have the time. Tell your husband how sorry you are and that you have stopped the relationship on your own. Let him know what you said in your post, you know it is wrong and you do not want it to continue.
Also, make an appointment to see a therapist. If your husband will not go, go yourself. It is important that your husband sees you working on this issue and trying to make it better. Try this link to find a therapist- http://www.bacp.co.uk/
Allow your husband to ask any questions he wishes to. It may be tough to hear the questions and answer them, but you need to let him in on what happened so he does not allow his imagination to make the situation worse.
If you attend church, consider talking with your pastor about the situation. Faith can do a lot to help a marriage recover from an affair.
Here are some other resources to help you:
Not "Just Friends": Rebuilding Trust and Recovering Your Sanity After Infidelity by Shirley P. Glass and Jean Coppock Staeheli
Unfaithful: Hope and Healing After Infidelity by Gary Shriver and Mona Shriver
How to Help Your Spouse Heal From Your Affair: A Compact Manual for the Unfaithful by Linda J. MacDonald M.S.
You can find these books on Amazon.com or your local library may have them for you.
With some work you and your husband can overcome this problem and build a stronger marriage.
I hope this has helped you,